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SHORT  RANGE  AIR  DEFENSE  (SHORAD)  ENGAGEMENT  PERFORMANCE  CRITERIA 


EXECUTIVE  SUMMARY 


Requirement: 

To  determine  operator  engagement  task,  and  summary  perfor¬ 
mance  parameters,  scenarios,  scenario  difficulty  scaling  factors, 
performance  measures,  performance  scoring,  and  performance  cri¬ 
teria,  and  test  administrative  procedures  for  applyinq  qualifi¬ 
cation  standaidt,  to  Military  Occupational  Specialty  (MOS)  16P, 
16S,  and  16R  soldiers.  The  required  research  products  combine  to 
form  the  Short  Range  Air  Defense  (SHORAD)  systems  performance 
criteria  and  applications  procedures. 


Procedure: 

The  draft  SHORAD  performance  criteria  were  determined  by 
means  of  the  analysis  of  field  test  and  experimental  data  col¬ 
lected  in  the  Realistic  Air  Defense  Engagement  System  (RADES) 
testbed  and  trainer.  A  scenario  difficulty  weighting  procedure 
was  employed  to  assess  agreement  between  empirically  defined 
scenario  difficulty  indexes  and  scenario  difficulty  ratings 
ascribed  by  subject  matter  experts. 


Findings : 

Results  obtained  from  field  test  experiments  and  the  RADES 
multi-experiment  database  were  used  to  determine  engagement  per¬ 
formance  standards  attainable  by  SHORAD  personnel.  Performance 
variations  were  consistent  with  expert  ratings  in  determining 
scenario  difficulty  levels.  Multidimensional  performance  cri¬ 
teria  and  scenario  difficulty  scales  were  determined  and  will  be 
subjected  to  empirical  validation. 

A  fully  integrated  and  automated,  scenario-specific  feedback 
and  roultiscenario  scoring  system  was  also  developed  and  tested. 
Over  200  training  and  test  scenarios  were  developed  and  indexed 
by  performance  difficulty  level.  Performance  criteria  cutoffs 
and  administrative  procedures  have  been  outlined,  implemented, 
and  tested,  in  anticipation  of  future  validity  testing. 


Utilization  of  Findings: 

The  proponent  for  this  research  was  the  Directorate  of 
Training  and  Doctrine  (DOTD) ,  United  States  Army  Defense 


Artillery  School  (USAADASCH) .  These  results  and  draft  standards 
were  briefed  to  the  Director,  DOTD,  on  26  September  1988.  This 
research  will  form  the  basis  for  draft  standards  of  performance 
for  SHORAD  crews  in  associated  gunnery  tables. 
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SHORT  RANGE  AIR  DEFENSE  ( 3HORAD )  ENGAGEMENT  PERFORMANCE  CRITERIA 


OVERVIEW 


operational  P r oblem 

At  the  request  of  the  Directorate  of  Training  and  Doctrine 
( DOTD )  of  the  US  Army  Air  Defense  Artillery  School  (USAADASCH), 
and  with  the  suport  of  the  US  Army  Missile  Command  -  Target 
Management  Office  (MICOM-TMO),  the  Army  Research  Institute  is 
developing  a  Range  Target  System  (RTS).  The  RTS  provides 
sustainment  training,  qualification,  and  certification  of  Short 
Range  Air  Defense  (3HORAD)  crews,  teams,  and  operators  in 
engagement  simulation  and  live  fire  exercises.  The  RTS  will  also 
support  future  Forward  Area  Air  Defense  System  ( FAADS ) 
components.  These  include  the  L  ine- of-S  ig  ht-Foward-'Ieavy  (LOS-F- 
H  )  ,  L ine- of-3 ig ht-Raar  (LOS-R),  and  Non-Line-of-S ight  (NLOS) 
weapons  system  and  the  Forward  Area  Air  Defense  Command,  Control, 
Communication,  and  Intelligence  { FAAD  C^I)  system. 

A  critical  component  of  the  RTS  will  be  the  range  tables, 
the  basis  of  which  will  be  derived  under  this  program  of 
research.  The  RTS  con f ig ura t i on  and  this  research  effort  are 
interdependent  projects.  The  RTS  promises  to  be  the  means  of 
i ns t i t ut iona 1 i z ing  range  table  procedures,  scenarios,  performance 
measures,  scoring,  and  performance  criteria  in  a  way  that  will 
improve  ADA  collective  crew  readiness  Armywide.  The  range 
tables  and  associated  RTS  are  expected  to  be  fielded  as  an 
integrated  product  to  ensure  an  objective  crew  sustainment  and 
qualification  program  of  tcainina  and  evaluation. 

The  product  of  this  research  will  be  valid,  reliable, 
practical,  and  economical  engagement  performance  range  tables  for 
use  in  5HDRAD  crew,  team,  and  operator  sustainment  training  and 
qualification  testing,  which  are  compatible  with  the  emerging 
Range  Target  System. 


Rase  a  r  £  h  _0  b  j_e  c  t  i  ve 

The  research  objective  is  to  determine  the  range  table 
components.  First,  test  and  training  scenarios  are  needed. 
Second,  performance  difficulty  level  indexes  are  needed  to 
partition  the  scenarios  for  training  and  test  purposes .  Third, 
engagement  performance  tusk  and  summary  performance  measures  are 
requirei  to  diagnose  the  errors.  Fourth,  engagement  performance 
criteria  (i.e.,  performance  standards)  are  needed  to  quality 
soldiers  and  to  establish  minimum  acceptable  levels  of 
performance.  Fifth,  range  table  administrative  procedures  and 
scenario  feedback,  across-scenar i o  scoring  algorithms,  displays, 
and  hardcopy  report  formats  are  needed  to  ensure  proper 
utilization  and  information  payoffs  from  the  application  of  the 
range  tables. 
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A£Qr oach 


The  approach  «ias  to  repeat  a  pre-defined  .sequence  of 
subtasks  until  the  research  objective  had  been  successfully 
attained.  Tnese  subtasks  were: 

•  Scenario  Scripting  and  Generation 

•  Scenario  Administration  Gonditions  Specification 

•  Summary  and  Task  Performance  Measures  Specification 

•  Scoring  Algorithn  Development 

•  Scenario  Feedback  Display  Definition 

•  Hardcopy  Mul t i- Scenar i o  Scoring  Report  Development 

•  Expert  Rating  of  Scenario  Difficulty  Factors 

•  Collection  of  Individual  Soldier  Differences  Data 

•  Field  Test  Data  Collection 

•  Multi-Experiment  Database  Management  and  Analysis 

•  Target  Parameters  and  Distribution  Specification 

•  Field  Test  Data  Analysis 

Soldiers  served  on  an  as  available  basis.  Subtasks  were 
performed  in  various  sequences  to  capture  meaningful  soldier 
performance  data.  Fortunately,  the  Realistic  Air  Defense 
Engagement  System  (RADES)  already  had  the  task  and  summary 
performance  measurement  capabilities  needed,  so  field  test 
performance  data  could  be  captured  and  stored  for  later  use  in 
range  table  development  and  verification.  The  RADES  testbed  was 
therefore  employed  to  meet  essential  data  requirements. 


RADES 


RADES  is  an  instrumented  testbed  and  SHORAD  engagement 
simulation  exercise  (ESX)  system.  SHORAD  soldiers  employing 
their  actual  weapon  systems  engage  scaled  fixed  wing  ( FW )  and 
pop-up  rotary  wing  (RW)  aircraft  in  an  outdoor  desert 
environment  under  controlled  field  test  conditions.  As  many  as 
five  weapons  and  their  associated  crews,  teams,  or  operators  can 
be  tested  or  trained  at  a  time.  Direct  weapon  connections 
a ut oma t ical 1 y  collect  interr oga t i on  friend  or  foe,  acquisition, 
track,  lock-on,  uncage,  superelevate ,  fire,  and  launch  signals 
from  the  weapon.  Detection,  identification,  and  command  to 
engage  or  cease  engagement  voice  commands  are  collected  using 
human  data  collectors  at  computerized  data  acquisition  stations. 
The  human  data  collectors  enter  event  data  via  the  data 
collection  station  keyboard. 
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.  ad.E.S  synthetically  flies  missile  rounds  to  target  intercept 
an.i  evaluates  the  outcome  as  a  kill  or  miss,  predicated  on  the 
status,  location,  and  range  of  the  aircraft  at  time  of  intercept. 
Time  delays  in  software  ensure  that  effects  are  not  provided  to 
the  weapon  tean  or  crew  until  the  intercept  would  have  occurred 
under  Cullscale  conditions.  RAPES  automatically  records  the 
reason  for  any  assessed  engagement  miss  (e.q.,  aircraft  out  of 
range,  failure  to  acquire,  no  lock-on  at  fire,  failure  to 
supe re  1 e v a ta ,  etc.).  The  RAPES  host  computer  provides  aircraft 
status,  location,  and  ran ja  data  to  the  data  collection  stations 
during  realtime  engagements.  The  host  also  collects  and 
consolidates  the  data  into  an  aggregate  test  file  at  the  ■>•!.!  >' 

every  engagement  trial.  It  is  this  aggregate  data  file  that  is 
returned  from  the  field  site  for  database  management  an  i 
a  n  a  1  /  s  i  s  . 


RAPE.!  research  has 
iects  on  air  iefense 


demonstrated  the  following  situational 
nance  (Rarber,  1987): 


•  Target  7  ha racteri sties 

-  Aspect,  elevation,  azimuth  (offset),  size,  type, 
and  range 

•  Weather  and  Terrain  Conditions 

-  Visibility,  weapon  position,  target  background 
'.contrast),  atmospheric  conditions  (wind,  temperature) 

•  individual  differences 

-  Sensory,  perceptual,  psychomotor,  cognitive,  and 
personal  i  f / 

•  level  of  Training  and  Scenario  difficulty 

-  Experience,  workload,  and  practice  level 

•  Command,  Control,  Communications,  and  Intelligence  (CJI) 

-  Alerting,  cuing,  reliability  of  information 

•  doctrine  and  Tactics 
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BACKGROUND 


In  nan/  ways  this  research  is  directed  at  correcting  current 
deficiencies  in  existing  engagement  performance  criteria, 
cr 1 te r i on- re f e rented  standards,  qualification  scenarios,  and  test 
administration  procedures.  Current  engagement  qualification  and 
training  criteria  and  standards  of  engagement  performance  are 
deficient  in  the  following  ways: 

•  Counterair  Effectiveness  (ordnance  delivery 
prevention  is  not  considered  by  existing 
scoring  procedures). 

•  E.£ie0.^iZ_A i r  (the  effects  on  performance  of 
complicating  friendly  air  elements  and 
corresponding  fratricide  rates  are  not  included 
in  current  qualification  and  certification 
testing  processes  and  standards  for  all  SHORAD 
systems) . 

•  Task  Difficulty  (the  significant  effects  of 
scenario-  and  envi ronment- imposed  task 
performance  difficulty  are  not  considered  in 
the  development  and  ordering  of  scenarios  for 
test  and  training  purposes). 

•  Achievable  Performance  Levels  (criteria  have 
not  been  developed,  nor  are  they  administered, 
with  consideration  of  achievable  soldier 
performance  levels) . 

•  Collective  Crew  Engagement  Performance  testing 
and  qualification  (current  standards  are 
directed  solely  at  gunner  part-task 
qualification  and  do  not  include  crew  chief, 
squad  leader,  or  team  chief  tasks). 

•  Test  Administration  Controls  and  Procedures, 
qualification  test  scenarios,  and  test 
conditions  (current  qualification  and 
certification  testing  procedures  are  subject  to 
a  large  amount  of  user  interpretation  and  vary 
considerably  in  test  administration  practices 
and  en v ir onmenta 1  conditions  from  one 
application  and  unit  location  to  another). 


To  overcome  these  weaknesses  in  current  engagement 
performance  qua  1 i f icat ion  and  certification  testing,  API ,  at  the 
request  of  the  Directorate  of  Training  and  Doctrine  (DOTP), 
formulated  a  comprehensive  approach  to  formal  range  table 
development . 

As  a  result,  the  present  research  has  given  rigorous 
attention  to  those  factors  not  normally  addressed,  or  only 
modestly  treated,  in  the  determinat ion  of  previous  qual i f icat i on 
standards  of  performance.  These  contributions  include  the: 


Crew,  team,  and  operator  distribution 
parameters  for  each  Summary  Performance  Measure 

Impact  of  using  a  weapon  control  status  of 
"tight"  versus  "free" 

Effect  of  introducing  multiple  targets 
simultaneously  or  sequentially 

Effect  on  Summary  Performance  Measure  scores  of 
introducing  friendly  aircraft 

Effects  of  including  the  squad  leader 

Impact  on  crew  performance  of  including  an 
ordnance  delivery  prevention  criterion 

Effects  of  various  types  of  alerting  and  cuing 

Consideration  of  soldier  capability  limitations 
which  affect  performance 


FINDINGS  AND  PERFORMANCE  CRITERIA  COMPONENTS 


This  section  describes  the  requirements ,  procedures,  and 
findings  for  all  the  subtask  actions  conducted  in  this  study. 
Specifically  addressed  are  the: 

•  Multi-Experiment  Database  Analysis 

•  Definition  of  Range  Table  Scenario  Test 
Conditions 

•  Range  Table  Scenario  Scripting  and  Scenario 
Generation 

•  Definition  of  Summary  Performance  Measures 
(SPM)  and  Task  Performance  Measures  (TPM) 

•  Scoring  Algorithm  Development 

•  Scenario  Feedback  Display  Definition 

•  Determination  of  Scenario  Difficulty  Factors 

•  Scenario  Field  Testing 

•  Determination  of  SPM  and  TPM  Cut-off  Scores 


The  report  includes  a  discussion  of  findings  directly 
addressing  the  future  actions  toward  fielding  of  the  3H0RAD 
sustainment  training  and  qualification  testing  range  tables. 


Multi-Experiment  Database  Management  and 


Requirement :  To  establish  baseline  performance  parameters 

for  Short  Range  Air  Defense  (SHORAD)  system  crews,  teams,  and 
operators,  and  to  generate  databases  on  human  performance  from 
which  to  draw  generalizations  about  the  ADA  population. 


Procedures:  Air  defense  engagement  part-task  and  summary 

performance  efficiency  and  effectiveness  were  measured  under  a 
wide  variety  of  environmental  and  scenario  test  conditions  using 
the  RADES  testbed  over  a  period  of  two  years.  Individual 
difference  measures  on  participating  soldiers  were  also  obtained 
coincident  with  those  experiments  contributing  to  the  RADES 
multi-experiment  database. 


Test  Conditions:  Analyses  were  based  on  prior  test  results 
obtained  from  RADES  experiments.  The  experimental  conditions 
existing  during  these  tests  were  clear  weather,  daylight 
conditions.  Data  reflected  the  performance  of  SHORAD  crews  and 
teams  reacting  to  l/7th  scale,  flying  fixed  wing  ( FW )  aircraft 
and  l/5th  scale,  pop-up  rotary  wing  (RW)  aircraft.  Results  from 
the  following  field  tests  were  consolidated  to  produce  a  meta¬ 
analysis. 
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•  Chaparral  Weapon  Control  Status  and 
Identification  Friend  or  Foe  (IFF)  Experiment 

•  Redeye  Weapon  Control  Status  and  Training 
Experiment 

•  Stinger  Terrain  and  Target  Characteristics 
Experiment 

•  Stinger  Early  Warning  and  Cuing  Experiment 

•  Tripod  Mounted  versus  Man-Portable  Stinger 
Experiment 

•  Enhanced  RADES  Observer  Experiment 

•  Stinger  Training  Experiment 


Table  1  presents  the  dependent  variables  used  in  the  meta¬ 
analysis.  Average  scores  for  these  variables  were  derived  based 
on  weighted  means  with  extreme  outliers  (i.e.,  +  or  -  3  standard 
deviations)  removed.  Average  scores  were  derived  by  taking  the 
mean  of  all  observations  for  a  single  crew  or  team  for  a  given 
scenario  type.  Therefore,  while  multiple  observations  were 
obtained  from  each  crew,  the  sample  sizes  reflect  the  total 
number  of  crews  participating,  not  the  total  number  of 
abservat ions . 

These  past  investigations  focused  upon  different 
experimental  manipulations  and  controls.  Hence,  the  results 
resented  in  this  section  should  be  viewed  as  the  overall, 
xpected  performance  of  the  3H0RAD  population  aggregated  across 
weapon  systems  and  experimental  conditions.  These  conditions 
inc 1 uded : 


•  Weapon  systems  (Stinger,  Chaparral,  and 
Redeye ) 

•  Experience  and  training  level  (trainees  to 
NCOs) 

•  Target  aspects  and  offsets  (face,  tail, 
side  views) 


and 


•  Target  approach  azimuth  and  flight  profile 

•  Target  background  (sky  versus  terrain) 

•  Early  warning  and  cuing  (varied  in  method, 
delay  time,  update  rate,  content,  and 
accuracy ) 

•  Alert  status  and  weapons  control  status 
("red  free",  or  "red  tight") 

•  Rotary  wing  range  (2.5  km  to  7  km, 
f ullscale ) 

•  Soldier  differences  (vision,  demographics, 
and  personality) 


Table  l 

Dependent  Variables 


j  CODE  |  TITLE  or  DESCRIPTION  |  DUTY  |  INTERPRETATION  j 

RDET 

Range  of  Detection 

SL  or 
SG 

The  slant  range  from 
the  weapon  to  the 
target  when  the  event 
took  place.  Range  is 
relevant  for  fixed 
wing  targets  only 
since  rotary  wing 
targets  simply 
popped-up  from  a 
static  position.  Ranges 
are  given  in  full  scale 
kilometers. 

RID 

Range  of  Identification 

SL 

RLOCK 

Range  of  Track  or 

Lock-on 

SG 

RFIRE 

Range  of  Weapon  Fire 

SG 

TDET 

Time  of  Detection 

SL  or 

SG 

Based  on  seconds  after 
target  availability 
where  availability 
begins  when  visual  line- 
of-sight  is  achieved. 

TID 

Time  of  Ident i f ica t ion 

H 

Time  interval  between 
Detect  and  Identification 

TL3CK 

Time  of  Lock-on 

SG 

Time  interval  between 
Detection  and  Lock-on. 

TFIRE 

Time  of  Weapon  Fire 

SG 

Time  interval  between 
Lock-on  and  Fire. 

THAND 

Time  of  Hand off 

Bot  h 

Time  interval  between 
Identification  and  Fire. 

TTOT 

Total  Engagement  Time 

Bot  h 

Time  interval  between 
Detection  and  Fire. 

IDCOR 

Correctness  of 

Identif ication 

SL 

Number  of  correct 
identifications  divided 
by  number  of  targets 
identif ied . 

PKILL 

Probability  of  Target 
being  engaged  and 
destroyed 

Bot  h 

Number  of  aircraft 
destroyed  divided  by 
number  presented. 

KEY:  SL  =  Team  or  Squad  Leader 

3G  =  Senior  Gunner 


1  n 


Findings:  Data  from  the  first  set  of  single  target  ( FW  and 

RW)  trials  from  several  RADES  tests  were  consolidated.  From 
these  consolidated  data,  engagement  parameters  were  estimated. 
The  performance  data  provided  herein  represent  overall  results 
drawn  from  two  years  (1985  and  1986)  of  RADES  experiments. 

Table  2  lists  summary  performance  results  for  the  3H0RAD 
population  for  FW  and  RW  targets  in  terms  of  overall  engagement 
outcomes.  Table  3  provides  FW  aircraft  ranges  in  full  scale 
kilometers  for  critical  engagement  outcomes  as  a  function  of 
aircraft  intent  (hostile  or  friendly).  Table  4  provides  these 
data  relative  to  aircraft  intent  for  each  model  type.  Table  5 
presents  the  approximate  RW  aircraft  event  times  in  seconds  for 
critical  engagement  outcomes  as  a  function  of  aircraft  intent. 
Table  6  presents  these  data  as  a  function  of  aircraft  intent  for 
each  model  type.  Blank  spaces  in  these  tables  indicate  that  the 
data  were  missing  or  the  sample  sizes  were  too  small  to  be 
meaningful. 

All  rotary  wing  event  times  were  based  on  the  time  when 
1 ine- of-s ig ht  (LOS)  was  first  established.  LOS  was  defined  as 
the  point  in  time  at  which  the  helicopter  rotor  blades  first 
broke  the  terrain  masking.  The  entire  helicopter  became  visible 
approximately  2  seconds  after  that. 


Engagement  Parameter  Estimates:  The  fixed  wing  engagement 
event  sequence  and  the  rotary  wing  engagement  event  sequence  have 
been  depicted  in  Figures  1  and  2,  respectively.  These  figures 
show  approximate  population  parameter  values  associated  with 
critical  engagement  part-task  events  by  target  type  ( FW  or  RW ) , 
and  illustrate  how  these  events  unfold.  The  greatest  potential 
for  improvement  in  engagement  performance  can  be  afforded  by 
increasing  the  range  at  which  aircraft  are  detected,  identified, 
and  acquired.  Smooth  tracking,  lock,  and  fire  task  actions  are 
rather  tightly  grouped  in  terms  of  their  time  and  range  of 
occurrence.  Thus,  if  the  range  of  detection,  identification,  and 
acquisition  are  extended,  the  subsequent  ranges  of  track,  lock, 
and  fire  will  result  in  greater  ranges  of  engagement. 

Figures  3  and  4  depict  the  overall  expected  distributions  of 
the  S HO RAD  population  with  respect  to  FW  and  RW  engagement 
events,  respectively  (assuming  a  normal  distribution).  Some 
sample  distributions  contributing  to  the  population  estimates 
were  skewed.  Therefore,  it  must  be  noted  that  the  population 
standard  deviations  may  be  lower  than  those  reported  here,  while 
the  means  are  believed  to  be  representative  of  the  population. 

The  recommended  criteria  were  approximated  from  various  source 
documents  which  specify  weapons  system  and  soldier  performance 
requirements  (US  Army  DCD ,  1987;  Headquarters,  Dept,  of  the  Army, 
1988).  These  criteria  are  indicated  in  Figures  1  and  2  with  an 
and  in  Figures  3  and  4  with  a  "C" .  The  criteria  indicated  in 
Figures  1-4  are  arbitrary,  and  do  not  represent  classified  data. 


Table  2 

SHORAD  Summary  Performance  Data 


FW  |  RW 


| SUMMARY  PERFORMANCE  MEASURES 

% 

N 

% 

N  | 

Correctness  of  ID 

75 

49 

85 

57 

Hostiles  Engaged 

77 

37 

81 

57 

Friends  Engaged 

39 

37 

23 

57 

Engaged  Aircraft  Destroyed 

92 

33 

85 

41 

Hostile  Attrition 

72 

31 

76 

41 

Friendly  Fratricide 

35 

31 

19 

41 

Hostiles  Releasing  Ordnance 

62 

37 

83 

41 

Table  3 

FW  Event  Ranges  (in  kilometers)  and  Performance  Outcomes 
by  Intent 


FRIENDLY  |  HOSTILE  |  OVERALL 

MEAN  j  SD  |  MEAN  |  SD  |  MEAN  |  SD 


Table  4 

FW  Event  Ranges  (in  kilometers)  and  Performance  Outcomes  by 
Model  Type 


FRIENDLY 

A 7  |  A 10  |  F 16  I  Fill 

MEAN]  SO j  N | MEAN |  SO |  N | MEAN j  SO | N  | MEAN |  SD |  N 
10.3|2.0|25|11.3|2.5|27|  8.1|2.2|44|13.4|4.0|18 


Table  5 

RW  Event  Times  { in  seconds)  and  Performance  Outcomes  by  Intent 


FRIENDLY 


HOSTILE 


OVERALL 


TDET 


TID 


TFIRE 


IDCOR 


PKILL 


j  MEAN 

SD 

|  MEAN  | 

SD 

1 

MEAN  | 

oo 

• 

-0 

2.5 

1  7-2  1 

3.5 

7 

8.3 

1  8-6  1 

4.1 

1  5-4  | 

3.0 

i 

7.0 

1 

1  2-8  1 

2.6 

7 

Table  6 

RW  Event  Times  (in  seconds)  and  Performance  Outcomes  by 
Model  Type 


1 

FRIENDLY 

I 

|  AH  1 

1 

UHl  ]  CH  3 

1 

j MEAN |  SD 

N| 

MEAN)  SD|N|MEAN|  SD 

1  N  | 

|  TDET 

|  3.2 

3.6 

40  | 

@9 

1.5|8| 

j  TIO 

j  8.2 

O' 

• 

40  | 

9.0  |  2.9  |9  j 

9 . 0 

4.6 (8 | 

| TFIRE 

|  2.2 

0.8 

15  1 

1 

1 

| THAND 

|  4.0 

1.5 

|19| 

1 

1 

j  TTOT 

6.5 

21  1 

1 

1 

| IDCOR 

1 

77% 

1 

87%  j 

75%  | 

j  PKILL 

1 

14% 

1 

1 

1 

j  HOSTILE  | 

|  MI  8 

|  MI  24 

j  MI  28 

1 

|MEAN|  SD 

N|MEAN|  SD  | 

N|MEAN  |  SD 

|N| 

|  TDET 

3.2 

1  43  | 

6.9 

4.0  |  44 j 

10.9  | 

3.4|9| 

j  TID 

1  5.4 

3.1 

|40| 

5.0 

3.3  j  41 j 

5-7  | 

2 . 0  |  9  | 

| TFIRE 

|  2.3 

1.6 

|39| 

3.3 

3.0  |  41 j 

2 . 4  | 

3 . 3  |  9  | 

| THAND 

|  4.8 

2.9 

|44| 

5.3 

4.1 | 44 [ 

9.4  | 

6.3j9| 

|  TTOT 

|  9.9 

3.0 

|44| 

9.9 

4. 5 [44  | 

| IDCOR 

1 

90% 

1 

94%  | 

87% 

1 

|  P  K I  L  L 

1 

88% 

1 

77%  | 

i 

Figure  1 

FW  Engagement  Event  Sequence 


Figure  2 

RW  Engagement  Event  Sequence 


Time  in  Seconds 


j  24*  22 

20  18  16  14 

12 

10  8  6 

Hi 

2  0  | 

Lock 

Det 

(16.2) 

A  , 

(3.0  ) 

A 

V  V 

Fire  ID 

V 

LOS 

(19.0)  (15.0) 

*  =  Criterion 


1  () 


c 


KEY :  Otarget  outbound 

Ocriterion 


Figure  3 


Consistency  of  Estimates  with  Full  Scale  Test  Results:  The 
parameter  estimates  presented  were  found  to  agree  with  data 
obtained  from  field  tests  conducted  using  full  scale  aircraft. 
Tables  7  through  13  present  comparisons  (when  experimental 
conditions  were  similar  enough  to  warrant  a  comparison)  and 
demonstrate  the  consistency  of  RADES  data  with  full  scale  field 
test  data.  Readers  are  also  referred  to  the  RADES  Validation 
Report  (Drewfs,  Barber,  Johnson,  and  Freder ickson ,  1988). 


Conclusions:  The  purpose  of  the  me t a-ana 1 ys i s  was  to 

consolidate  similar  studies  which  investigated  the  same 
phenomena,  in  order  to  make  generalizations  about  the  5H0RAD 
soldier  population.  The  findings  reported  are  assumed,  for 
purposes  of  range  table  development,  to  be  the  best  available 
a ppr o < ima t i on  of  the  true  population  parameters  based  upon  an 
aggregation  of  a  series  of  RADES  experiments  (3arber,  1987). 

A  major  conclusion  of  the  meta-analysis  was  that  the 
detection,  identification,  and  acquisition  of  aircraft  consume 
the  majority  of  the  engagement  process.  Improvements  in 
detection  and  identification  range  and  accuracy  would  result  in 
major  jains  in  overall  3H0RAD  engagement  efficiency  and 
effectiveness.  Some  gains  could  also  be  found  by  selecting 
soldiers  with  superior  vision,  and  providing  the  fire  unit  with 
accurate,  timely,  and  consistent  early  warning  and  cuing 
information.  Use  of  RADES  or  RTS  for  training  and  qualification 
testing  and  troop  unit  sustainment  training  would  increase 
hostile  attrition  and  fratricide  avoidance  at  optimal  ranges. 

This  is  because  it  enables  soldiers  to  practice  and  master 
aircraft  identification  and  engagement  skills,  against  friendly 
and  hostile  targets,  in  a  realistic  range  environment. 

Another  important  conclusion  was  that  many  soldiers 
currently  appeared  to  fall  below  the  performance  criteria  for 
fixed  wing  engagements;  only  half  the  population  met  the 
criterion.  Again,  improvement  could  likely  be  acquired  by 
reducing  the  time  and  increasing  the  range  of  detection, 
iden t i f i ca t i on ,  and  acquisition.  While  RW  performance  was 
consistently  above  criterion,  the  criterion  failed  to  consider 
helicopter  ordnance  release  prevention.  Assuming  an  ordnance 
delivery  time  of  18  to  19  seconds  for  a  helicopter  popping-up  and 
hovering  at  a  range  of  3  to  4  kilometers,  half  the  population 
would  be  above  and  half  below  this  adjusted  criterion,  indicating 
a  situation  similar  to  that  found  for  fixed-wing. 

It  is  also  important  to  note  that  the  ineta-analysis  only 
considered  single  target  presentations,  and  did  not  cover  the 
effects  of  multiple  or  sequential  targets.  Previous  research  in 
RADES  has  often  shown  poorer  performance  in  conditions  with 
multiple  targets  than  in  conditions  with  single  targets. 


Table  7 

Fixed  Wing  Event  Range  Comparisons  (in  kilometers) — 
Wright  (1966)  versus  RADES  (1987) 


RADES  ||  WRIGHT | |  COMPARISON 


|  VAR 

|  Mean 

3D |  N | | Mean 

N| 

| t-val ue 

|df |2- 

tailed  p| 

j  RDET 

|  10.8 

3. 2 | 52  |  | 10.0 

27  | 

|  1.05 

l77l 

p>.  2  | 

|  RID 

|  5.8 

2. 3  |  52 |  |  6.8 

27  | 

|  1.82 

l77l 

p> . 05  | 

NOTE:  Assumes  equal  variances 


Table  8 

Fixed  Wing  Event  Range  Comparisons  (in  kilometers)  for  the  A7 
and  A 10 — Tillapaugh  &  Smith  (1983)  versus  RADES  (1987) 


I 

1 

A7 

II 

A10  | 

|  VAR 

| t- value 

df | 2- ta i led 

p |  |  t_-value 

df 

|  2-tailed  p| 

|  RID 

|  1.0 

38  |  p>  .  2 

II  0.i0 

56 

|  P>.2  | 

| RFIRE 

|  1.5 

38 |  p>  .  2 

II  2-31 

49 

J  .01 <p< .05  | 

NOTE :  RADES  "fire"  value  based  on  data  from  hostile 

aircraft  only 


Table  9 

Friendly  Rotacy  Wing  Event  Time  Comparisons  (in  seconds) — 
Lott  (1977)  versus  RADES  (1987) 


|  Time  Interval 

:  Detection 

to  Identification  j 

j  t-val ue 

1 

2-tailed  p  | 

|  0.10 

1 

P>.  2  | 

NOTE:  Assumes  equal  variances 


Table  10 

Helicopter  Event  Time  Comparisons  (in  seconds)  for  the  AH  1  — 
CDEC  (1978)  versus  RADES  (1987) 


1 

RADES 

|  J  HAT 

(3-4 

km) 

|  |  HAT 

(2-6 

km) 

1 

|  VAR 

|  Mean 

SD  | 

N 

|  |  Mean 

3D 

N 

| | Mean 

SD 

N 

1 

|  TDET 

3.6  | 

40 

II  s-5 

8.1 

24 

||10.5 

9.1 

32 

1 

j  TID-DET 

rvj 

• 

CO 

4.9  | 

40 

12.1 

24 

II  6.0 

8.6 

32 

1 

|  TID* 

1  16.4 

4.9  | 

40 

|  |  16.0)12.7 

24 

||16.5 

11.6 

32 

1 

*Indicates  time  from  line-of-sight  to  identification  response 


j  RADES 

vs.  HAT 

( 3-4  km) j 

|  RADES 

vs.  HAT 

( 2-6  km) | 

|  VAR 

j  t-value 

1  a£  |2* 

tailed  pj 

| t-value 

1  d£  I2- 

tailed  p| 

|  TDET 

)  .89 

1  62  | 

P>.2  ) 

nis 

P>.1  | 

| TID-DET 

1  *79 

1  62  1 

p>.  2  j 

|  0.25 

1  20  | 

P>  •  2  | 

j  TID* 

|  .18 

1  62  | 

p>.  2  | 

|  0.05 

1  70  ) 

p>.  2  ) 

*Indicates  time  from  line-of-sight  to  identification  response 
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The  final  observation  that  warrants  discussion  relates  to 
assessed  aircraft  kills.  The  rate  of  fratricide  was  somewhat 
high,  and  unacceptable  to  the  friendly  air  community  (19%  and  35% 
in  Table  2;  14%  in  Table  6).  The  attrition  rate  on  hostiles  was 
below  the  established  limits  (military  standards  require  75%  and 
the  achieved  level  was  60%  to  70%).  This  reflects  a  need  to 
include  the  requirement  to  discriminate  friendly  and  hostile  air 
elements  in  training  as  well  as  in  qualification  and 
certification  testing. 


Definition  of  Scenario  Test  Conditions 

Requirement :  To  define  live  fire  exercise  (LFX)  and 

engagement  simulation  exercise  (ESX)  test  conditions  and  aircraft 
model  inventory  requirements  needed  for  the  proper  administration 
of  3H0RAD  training  and  qualification  testing  scenarios.  Test 
conditions  and  aircraft  model  specifications  must  insure  an  equal 
probability  of  successful  performance  across  weapons,  crews, 
teams,  and  operators,  within  each  particular  difficulty  level 
grouping  of  scenarios. 


Procedures ;  Scenario  test  conditions  were  profiled  for  both 
engagement  simulation  and  live  fire  test  purposes.  Differences 
in  live  fire  range  safety  and  testing  procedures  mandated  that 
the  two  test  conditions  be  specified  separately.  Draft  test 
conditions  specifications  were  then  provided  to  ARI 
representatives  for  review  and  comment.  Comments  received  back 
from  ARI  were  then  used  to  adapt  the  draft  test  conditions 
specifications. 


Findings:  Table  11  provides  recommended  specifications  for 

test  conditions,  and  Table  12  lists  recommended  aircraft  model 
spec i f icat ions  for  the  LFX  and  ESX  engagement  range  tables. 

While  live  fire  specifications  may  change,  given  developments  in 
live  fire  procedures,  less  change  is  expected  in  ESX  test 
conditions  specifications.  Figure  5  presents  the  basic  ESX  range 
layout  utilized  during  field  tests  associated  with  criteria 
development  and  validation  efforts. 


Scenario  Scripting  an d  Gene ration 

Requirement :  To  establish  a  library  of  engagement 

qualification  and  training  scenarios  for  use  in  administering  the 
3H0RAD  range  tables.  The  scenario  library  must  be  inclusive  of  a 
full  range  of  factors  known  to  alter  SHORAD  part-and  whole-task 
performance  difficulty  (e.g.,  alerting  and  cuing,  target  size, 
target  type,  target  number,  and  target  intent). 


•>  > 


Table  11 

Recommended  Test  Conditions 


Engagement  S imulation  (ESX) 

•  Sky  Background 

•  Clear  Day  (20  +  Miles 

Visibility) 

•  Stationary  Weapon 

Position 

•  93  degree  Search 

Sector 

•  Unaided  Detection 

•  Aided  Recognition 

(binoculars) 

•  Cuing  (+  or  -  15  degrees 

accuracy ) 

•  Early  Warning  Voice 

Message 

(50  seconds  prior  to 
availability) 

•  Air  Defense  warning 

Red,  WCS  Tight 

•  IFF  Return  Unknown 

•  One  RW  Practice  Trial 

•  No  trial-by-trial 

Feedback  (End  of  day 
feedback ) 

•  No  Visitors  at  Weapon 

Site 

•  Windspeed  not  to 

Exceed  25  MPH 

•  Randomized  Scenario 

Order 

•  Standardized  Scenario 

Set 

•  Standard  Target 

Coloration 

•  Matched  Target  Sizes 

•  RW  Range: 

Stinger  =  2  to  5  Km; 
Chaparral  =  2  to  5  Km; 
Vulcan/PIVADS  =  .5  to 
1  KM 

•  FW  Availability  - 

20  to  33  Km 

•  FW  Airspeed  - 

33  to  100  MPH 
(1/5  Scale) 

•  4-Hour  Blocks 


Live  Fire  (LFX) 

•  Sky  Background 

•  Clear  Day  (23+  Miles 

Visibility) 

•  Stationary  Weapon 

Position 

•  90  degree  Search 

Sector 

•  Unaided  Detection 

•  Aided  Recognition 

( binoculars ) 

•  Cuing  (+  or  -  15 

degrees  accuracy) 

•  Early  Warning  Voice 

Message 

(60  seconds  prior  to 
availability) 

•  Air  Defense  warning 

Red,  WCS  Free 

•  IFF  Return  Unknown 

•  One  RW  Practice  Trial 

•  No  trial-by-trial 

Feedback  (End  of  day 
feedback ) 

•  No  Visitors  at  Weapon 

Site 

•  Windspeed  not  to 

Exceed  25  MPH 

•  Randomized  Scenario 

Order 

•  Standardized  Scenario 

Set 

•  Standard  Target 

Colorat ion 

•  Matched  Target  Sizes 

•  RW  Range: 

Stinger  =  2  to  5  Km; 
Chaparral  =  2  to  5  Km; 
Vulcan/PIVADS  =  . 5  to 
1  KM 

•  FW  Availability  - 

10.5  Km 

•  FW  Airspeed  - 

80  to  103  MPH 
(1/5  Scale) 

•  2-Hour  Blocks 


Qj  O 


Table  12 

Recommended  Aircraft 


j  FIXED  WING 

ROTARY  WING  j 

|  Friendly  J  Hostile 

Friendly 

Hostile  | 

A10  Thunderbolt 

A7  Corsair 

F16  Falcon 

Fill  Strike-Bomber 

MIG27  Flogger 
SU17  Fitter 
SU24  Fencer 
SU25  Frogfoot 

UH1  Iroquois 
UH60  Blackhawk 
AH64  Apache 

CH3  Green  Giant 

MI 8  Hip 

MI24  Hind 

MI 28  Havoc 
MI?  Hokum 

Targets  that  ace  "similar"  in  size 


•  A7  and  Fib  and  SU25 

•  A10  and  MIG27  and  SU17 

•  Fill  and  SU24 

•  UH60  and  AH64 

•  CH3  and  MI24  and  MI28 


Procedures:  RADES  scenario  scripting  software  was  used, 

along  with  the  DBASE  III  Plus  relational  database  management 
software  package,  to  establish  the  initial  set  of  20  scenarios. 
Scenario  scripts  were  then  produced  for  a  basic  library  of  200 
scenarios.  Given  approval  of  the  200  scenarios  by  ARI ,  the  200 
scenarios  will  be  encoded  for  use  in  the  RTS.  The  20  baseline 
scenarios  were  used  as  the  test  stimulus  in  SHORAD  range  table 
tryout  testing. 


Findings:  Appendix  A 

for  air  defense  range  tabl 
While  these  are  draft  seen 
to  preserve  the  scenario  g 
onstruction.  In  addition 
ifficulty  is  provided  in 
consideration  those  factor 
recent  field  data  collecti 
workload . 


provides  the  200  developed  scenarios 
e  qualification  and  training  purposes, 
arios,  SAIC  has  taken  considerable  care 
oals  of  DOTD  and  ARI  in  their 
,  an  SAIC  estimate  of  situational 
the  table,  which  takes  into 
s  shown  by  the  RADES  meta-analysis  and 
on  efforts  to  affect  performance  and 


Summary  and  Task  Pe rf ormance  Me a sures  Definition 

Requirement :  To  define  summary  performance  measures  (SPM) 

which  discriminate  qualified  crew,  team,  and  operator  engagement 
performance  from  unqualified  performance: 

•  Under  a  wide  range  of  scenario  difficulty 
conditions. 


:  i 


•  Under  a  wide  range  of  environmental  conditions, 

•  Across  the  existing  family  of  SHORAD  weapon 
systems  (i.e.,  Vulcan  or  PIVADS,  Chaparral,  and 
Stinger ) , 

•  Remaining  sensitive  to  the  individual 
differences  of  sub-groups  of  soldiers  comprising 
the  SHORAD  soldier  population. 


A  second  requirement  was  to  define  part-task  performance 
measures  (TPM )  which  pointed  to  the  sources  of  crew,  team,  or 
operator  5PM  failure,  and  which  could  be  used  validly,  reliably, 
practically,  and  economically  to  assign  corrective  training 
actions : 


•  Across  difficulty  levels  of  scenarios, 

•  Across  environmental  conditions, 

•  Across  the  existing  family  of  SHORAD  weapon 
systems. 


Procedures :  SPM  and  TPM  were  selected  if  they  had  been 

shown  in  prior  field  test  experiments  to  separate  crews,  teams, 
or  operators  on  the  basis  of  variation  in  performance.  The 
measures  sought  were  those  sensitive  to  performance  efficiency 
(speed,  or  TPM)  or  effectiveness  (accuracy,  or  SPM),  and  of  value 
in  distinguishing  between  qualified  and  unqualified  crews,  teams, 
and  operators.  In  addition,  measures  useful  in  diagnosing  the 
sources  of  unqualified  performance  were  also  identified.  In  this 
regard,  SPM  and  TPM  which  were  selected  demonstrated  substantial 
variance.  The  ones  that  didn’t  were  those  for  which  performance 
was  virtually  perfect  (95%  or  above),  and  they  were  not  selected. 

Table  13  lists  all  summary  performance  measures  and 
definitions  developed  under  the  present  research  program.  In 
addition,  under  LFX  data  collection  operations,  the  SPM  of  number 
of  rounds  in  the  target  area,  mean  hit  point,  and  average  miss 
distance,  were  integrated  into  the  SPM  set.  Table  14  lists  all 
task  performance  measures  and  definitions  developed  under  the 
present  research  program.  These  candidate  TPM  and  SPM  were 
reduced  after  demonstrating  which  ones  did  not  discr iminate 
levels  of  achievement  effectively. 


LiQ^iO.221  What  was  sought  in  this  analysis  was  maximum 
discrimination  between  groups  on  the  basis  of  performance  level. 
TPM  which  did  not  contribute  to  performance  discrimination  were 
dropped  from  further  consideration.  For  example,  fixed  wing 
event  response  times  were  shown  in  prior  RADES  research 
applications  to  contribute  little  to  the  discrimination  of  high, 
medium,  and  low  performers  while  fixed  wing  event  ranges  were 
extremely  useful.  Thus,  only  the  event  ranges  were  considered 
essential  as  FW  TPM.  The  summary  performance  measures 
recommended  for  elimination  were  percent  targets  detected  and 
percent  aircraft  identified.  Both  of  these  measures  tended  to 
exceed  95%  in  the  RADES  experiments  conducted  to  date.  These 
\,ai.iables  should  not  be  confused  with  "percent  targets  correctly 
identified",  or  "time  and  range  of  detection  and  identification" 
as  these  are  among  the  most  important  measures  studied  within  the 
present  research  effort.  Further,  the  interrogation  (IFF)  event 
was  found  to  be  ineffective  in  discriminating  achievement  level, 
and  was  recommended  for  elimination  from  the  criterion  set. 
Instead,  IFF  was  recommended  for  use  as  a  teaching  point,  as  this 
event  should  occur  as  soon  as  the  target  has  been  detected. 
Finally,  the  range  and  time  of  command  to  engage  or  cease 
engagement  were  found  to  correlate  very  highly  with  the 
identification  event,  so  only  the  identification  ranges  and  times 
were  recommended  for  use  as  criteria. 


Scoring  A 1  gorithm  Develo pinent 


Requirement :  To  establish  a  free-standing  software  package 

compatible  with  the  Range  Target  System.  The  package  must 
transform  raw  exercise  data,  calculate  TPM  and  3PM  scores, 
compare  the  calculated  scores  with  TPM  and  SPM  criterion  cut-off 
values,  and  output  hardcopy  reports.  Hardcopy  reports  must  be 
produced  for  each  respective  crew,  team,  and  operator,  scenario 
performance  difficulty  level,  and  aircraft  type  class  ( RW  or  FW ) , 
such  that  only  "like"  scenarios  scores  are  reported  within  any 
one  hardcopy  report  for  each  crew. 


Ph  o 


able  13 

andidate  Summary  Performance  Measures  (SPM) 


CODE  |  EVENT  |  DESCRIPTION  |  DUTY 


PDET 

Proportion  of 
Aircraft  Detected 

Number  of  detections 
divided  by  presentations 

SL  &  SG 

PID 

Proportion  of 
Aircraft  Identified 

Number  of  identifications 
divided  by  presentations 

SL 

IDCOR 

Correctness  of 
Identifications 

Number  of  correct  IDs 
divided  by  presentat ions 

SL 

j  FIDCOR 

Friendly 

Identifications 

Number  of  correct  IDs 
divided  by  presentations 

SL 

HIDCOR 

Hostile 

Identifications 

Number  of  correct  IDs 
divided  by  presentations 

SL 

ENGAGE 

Aircraft  Engaged 

Number  of  engagements 
divided  by  presentations 

SL  Sc  SG 

!  FENG 

1 

Friendlies  Engaged 

Number  of  engagements 
divided  by  presentations 

SL  &  SG 

HENG 

Hostiles  Engayed 

Number  of  engagements 
divided  by  presentations 

SL  &  SG 

FRAT 

Friendly  Fratricide 

Number  of  friendly  kills 
divided  by  presentations 

SL  &  SG 

| ATTRIT 

1 

Hostile  Attrition 

Number  of  hostile  kills 
divided  by  presentations 

SL  &  SG 

EFFECT 

Engaged  Aircraft 
Destroyed 

Number  of  kills  divided 
by  engagements 

SL  &  SG 

ORD 

Hostiles  Releasing 
Ordnance 

Number  of  ordnance 
releases  divided  by 
hostile  presentations 

SL  &  SG 

Table  14 

Candidate  Task  Performance  Measures  (TPM) 


|  CODE 

j  EVENT 

1 

DESCRIPTION 

DUTY  | 

j  RDET 

| Detection 

j  FW 

slant 

range 

at 

detect  ion 

SL  &  SG | 

|  RACQ 

| Acquis  it  ion 

|  FW 

slant 

range 

at 

acquisition 

SG  | 

j  RIFF 

1 Interroqat ion 

I  FW 

slant 

range 

at 

interrogation 

SG  1 

j  RID 

I  dent i f icat ion 

FW  slant  range  at  identify 

SL 

1 

j  RSNG 

Command  Engage 
or  Cease  Engage 

FW  slant  range  at  command 
engage  or  cease  engagement 

SL 

|  RLOCK 

Lock- on 

FW  slant  range  at  lock- on 

SG 

1 

|  RFIRE 

Fire 

FW  slant  range  at  fire 

SG 

I 

RHAND 

Hand- of  f 

FW  range  interval  from  identify 
to  fire 

r*  *■  r 

SG 

RTOT 

Total 

FW  range  interval  from  detect 
to  fire 

SL  & 

SG 

TDET 

Detection 

RW  time  interval  from  LOS  to 
detection 

SL  & 

SG 

j  TACQ 

Acquisition 

RW  time  interval  from  detect 
to  acquire 

SG 

TI  FF 

Interrogation 

RW  time  interval  from  detect  to 
IFF 

SG 

T  ID 

Iden  t i f ica  t ion 

RW  time  interval  from  detect  to 
identify 

SL 

TSNG 

Command  Engage 
or  Cease  Engage 

RW  time  interval  from  identify 
to  command  engage  or  cease 
engagement 

SL 

TLQCK 

Lock- on 

RW  time  interval  from  acquire 
to  lock-on 

SG 

TFIRE 

Fire 

RW  time  interval  from  lock-on 
to  fire 

SG 

THAND 

Hand- of  f 

RW  or  FW  time  interval  from 
identify  to  fire 

SL  & 

SG 

TTOT 

Total 

RW  or  FW  time  interval  from 
detect  to  fire 

SL  & 

SG 

'  q 


F  r  occ  dures .  The  research  team  employed  RADES  field  test 
data  as  the  example  input.  These  files,  containing  raw  field 
data  from  prior  experiments,  were  fed  into  a  prototype  score 
calculation  procedure.  This  resulted  in  associated  TPM  and  SPM 
scoring  outputs.  Next,  a  pass-fail  determination  routine 
associated  with  SPM  scores  was  generated,  based  on  realistic 
qualification  cut-off  values.  Then  the  TPM  diagnostics 
calculation  procedure  was  developed,  which  would  indicate  which 
TPM  contributed  most  to  whether  SPM  criteria  were  met  or  not. 
Utilities  were  added  for  the  purposes  of  calculating  scores  for 
all  crews  associated  with  a  multi-station  test  facility, 
specifically  the  multiple-weapon  RADES  configuration.  New  test 
data  collected  using  the  RADES  testbed  were  brought  in  from  the 
field  on  floppy  disk,  and  fed  into  the  newly-developed  software 
scoring  system  to  assess  the  degree  of  consistency  with  prior 
results.  This  was  the  final  test  and  calibration  step  in  the 
procedure  for  algorithm  development. 


Findings :  After  prototype  development  and  testing,  the 

scoring  transf ormat ion  algorithms  and  software  implementing  those 
algorithms  were  verified  as  operational.  Table  15  provides  the 
final  score  transformation  algorithms  and  Table  16  depicts  an 
example  of  the  hardcopy  report  output  for  RW  time  data.  Not 
shown  in  the  algorithm  figure  is  the  filter  used  to  deal  with 
measures  whose  values  are  missing  due  to  invalid  engagement 
sequences,  or  equipment  ma 1 f unc t ions .  This  software  filter 
automatically  prevents  the  miscalculation  of  scores  due  to 
missing  values.  (While  the  scoring  system  is  currently  written 
in  DBASE  III  Plus  command  language,  it  is  anticipated  that  this 
software  will  be  either  compiled  into,  or  translated  into,  the 
"C"  programming  language  in  order  to  increase  execution  speed. 
That  translation  will  be  part  of  the  RTS  integration  and 
demonstration  program). 


Scenario  Feedback  Display  Defin i t i on 

Requirement :  To  establish  scenario- specific  performance 

standards  to  be  displayed  as  feedback  which  can  be  used  in  the 
sustainment  training  of  SHORAD  crews,  teams,  and  operators.  The 
scenario  feedback  display  system  is  meant  to  be  integrated  into  a 
field  testing  and  sustainment  training  facility  such  as  RADES  or 
the  RTS.  It  is  currently  anticipated  that  the  feedback  displays 
will  not  be  used  during  SHORAD  qualification  and  certification 
test  exercises  but  will  be  used  exclusively  for  training 
purposes.  Use  of  the  feedback  displays  in  association  with 
testing  could  alter  the  performance  of  tested  crews,  teams,  or 
operators  and  result  in  invalidation  of  the  performance 
standards. 


Procedures :  A  mod i f ica t i on  of  the  previous  RADES  feedback, 

display  system  Tsee  Figures  6  and  7)  was  used  as  the  prototype 
for  future  SHORAD  training  feedback.  The  major  alteration  made 
to  this  ARI-approved  and  field-tested  display  (not  shown  in 
figures)  was  the  addition  of  task  performance  measure  cut-off 
values  (criteria),  so  that  not  only  actual  but  required 
performance  can  be  seen  by  the  exercising  soldiers  and  their 
instructors.  These  criterion  values  could  then  be  compared 
easily  to  achieved  soldier  performance. 


Findings:  Figures  6  and  7  provide  examples  of  the  SHORAD 

crew,  team,  and  operator  feedback  display.  It  is  important  to 
distinguish  the  feedback  display,  which  is  scenario-specific, 
from  the  scoring  system  and  hardcopy  reports,  which  cover  all 
scenarios  administered  to  a  crew,  team,  or  operatoc  in  the  course 
of  a  qualification  and  certification  test  or  training  exercise. 
While  feedback  displays  are  generated  one  per  trial  for  training 
purposes,  hardcopy  scoring  reports  are  generated  for  training  or 
test  purposes,  after  several  or  all  of  the  scenarios  have  been 
administered . 


Table  15 

Scoring  Transformation  Algorithms  for  RW  Time  Data 


1.  If  time  of  interrogation  friend  or  foe  greater  than  or 
equal  to  time  of  detection,  transformed  time  of 
interrogation  friend  or  foe  equals  time  of 
interrogation  minus  time  of  detection. 

2.  If  time  of  identification  greater  than  or  equal  to  time 
of  detection,  transformed  time  of  identification  equals 
time  of  identification  minus  time  of  detection. 

3.  if  time  of  acquire  greater  than  or  equal  to  time  of 
detection,  transformed  time  of  acquire  equals  time  of 
acquire  minus  time  of  detection. 

4.  If  time  of  command  to  engage  greater  than  or  equal  to 
time  of  identification,  transformed  time  of  command  to 
engage  equals  time  of  command  to  engage  minus  time  of 
identification. 

5.  if  time  of  command  to  cease  fire  greater  than  or  equal 
to  time  of  identification,  transformed  time  of  command 
to  cease  fire  equals  time  of  command  to  cease  fire 
minus  time  of  identification. 

6.  If  time  of  lock-on  greater  than  or  equal  to  time  of 
acquire,  transformed  time  of  lock-on  equals  time  of 
lock-on  minus  time  of  acquire. 

7.  If  time  to  superelevate  is  greater  than  or  equal  to 
time  of  command  to  engage,  transformed  time  to 
superelvate  equals  time  to  superelevate  minus  time  of 
command  to  engage. 

8.  If  time  of  fire  greater  than  or  equal  to  time  of  lock- 

on,  transformed  time  of  fire  equals  time  of  fire  minus 

time  of  lock-on. 

9.  If  time  of  fire  greater  than  or  equal  to  time  of 
command  to  engage,  transformed  time  of  hand-off  equals 
time  of  fire  minus  time  of  command  to  engage. 

10.  If  time  of  re-attack  (second  fire)  greater  than  or 

equal  to  time  of  fire,  transformed  time  of  re-attack 
equal  time  of  re-attack  minus  time  of  fire. 

11.  If  time  of  kill  greater  than  or  equal  to  time  of  fire, 

transformed  time  of  kill  equal  time  of  kill  minus  time 
of  fire  (i.e.,  round  flight  time).  Round  flight  time 
computations  were  based  on  approximations  of  classified 
data  to  protect  their  sensitivity. 


Table  16.  Scoring  Algorithm  Output 
FIXED  WING  SCENARIO:  DIFFICULT Y=HIGH 
TASK  PERFORMANCE  MEASURES  DIAGNOSTICS 

***********  k ************************  *************************** 


TPM 

MEAN 

STATUS 

CRITERIA 

RANGE 

OF 

DETECTION 

11768 

MEETS 

CRITERION 

8000 

RANGE 

OF 

ID 

3909 

BELOW 

CRITERION 

4000 

RANGE 

OF 

ACQUISITION 

3389 

BELOW 

CRITERION 

5000 

RANGE 

OF 

LOCK-ON 

2611 

BELOW 

CRITERION 

4000 

RANGE 

OF 

FIRE 

1760 

BELOW 

CRITERION 

2000 

SPECIAL  GUN  SYSTEM  LFX  SCORES  MEANS 

NUMBER  OF  ROUNDS  ON  TARGET 
MISS  DISTANCE 
HIT  POINT 

PASS-FAIL  DETERMINATION 

*************************************************************** 


3PM 

SCORE 

STATUS 

CRITERIA 

% 

CORRECT  ID 

100 

PASSING 

70 

% 

AC  DESTROYED 

100 

PASSING 

60 

% 

FRIENDS  ENG 

0 

PASSING 

30 

% 

HOST I L  SS  ENG 

100 

PASSING 

75 

% 

FRIENDS  CORRECT  ID 

100 

PASSING 

70 

% 

HOSTILES  CORRECT  ID 

100 

PASSING 

75 

% 

FRATRICIDES 

0 

PASSING 

25 

% 

ATTRITION 

100 

PASSING 

45 

SPECIAL  GUN  SYSTEM  LFX  SCORES  MEANS 

AVERAGE  NUMBER  OF  ROUNDS  ON  TARGET  = 

AVERAGE  NUMBER  OF  ROUNDS  PER  BURST  = 

AVERAGE  NUMBER  OF  TARGETS  KILLED 


REW:  01 


03/03/88 


14:56:26 


Figure  6 

Fixed  Wing  Scenario  Feedback  Screen 
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Figure  7 

Rotary  Wing  Scenario  Feedback.  Screen 
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Determination  of  Scenario  Difficulty  F actors 

Requirement To  determine  the  difficulty  of  a  scenario 
based  on  target,  soldier,  and  enviromental  factors. 


Procedures :  Difficulty  criteria  were  identified  based  on 

air  defense  research  findings  and  the  human  factors  literature. 
The  level  of  difficulty  criteria  are  given  in  Appendix  B.  Each 
difficulty  factor  was  assigned  a  relative  weight,  based  upon  its 
expected  effect  on  performance.  Three  sources  of  data  were 
utilized  for  obtaining  the  difficulty  scaling  factors  and 
ascribing  relative  weights  to  them.  These  were:  subject  matter 
experts,  the  available  literature,  and  previous  RADES  data. 

The  difficulty  level  for  each  scenario  in  a  standard  set  of 
20,  was  derived  by  each  expert  judge,  and  across  expert  judges. 
Agreement  among  the  experts  was  good  regarding  the  factors 
considered  important  for  establishing  scenario  performance 
difficulty  levels.  In  rating  the  difficulty  of  the  scenarios, 
interrater  reliability  was  also  good  as  demonstrated  by  Spearman 
correlations  (range  of  r-values  =  .17  to  .92;  average  r-value  = 
.68,  £  <  .001).  Subject  matter  experts  were  members  of  the 
research  team  from  ARI  and  SAIC. 

Data  (3PM  and  TPM)  obtained  from  field  test  administrations 
of  the  standard  set  of  scenarios  were  established  as  baseline 
parameters  of  performance.  The  difficulty  criteria  weights  were 
then  used  to  predict  variations  in  performance.  Performance 
variations  were  consistent  with  the  difficulty  criteria  weights 
ascribed  to  the  scenarios. 

The  method  used  to  derive  scenario  difficulty  scores  was 
based  on  classic  decision  analysis  logic.  The  process  began  with 
the  identification  of  difficulty  factors.  These  factors  were 
ascribed  relative  weights  by  subject  matter  experts  based  on 
their  evaluation  of  how  performance  was  affected  by  the  factors. 
Each  factor  was  comprised  of  several  subfactors  which  related  to 
specific  scenario  conditions  characteristic  of  the  factors  under 
which  they  were  categorized.  These  subfactors  were  ascribed 
values,  which,  when  multiplied  by  the  associated  factor  weight, 
resulted  in  a  difficulty  score  for  that  factor.  The  factor  score 
was  therefore  dependent  on  the  subfactor  which  was  relevant  to 
the  scenario  being  scaled.  A  total  difficulty  score  was  derived 
for  each  scenario  based  on  the  sum  of  all  difficulty  scaling 
factor  scores.  Scenario  difficulty  levels  (1  through  5,  or  low 
to  high)  were  assigned  to  each  scenario  which  reflected  the 
relative  differences  between  scenario  difficulty  scores. 


Findings:  Given  that  interrater  reliability  was  good, 

ratings  accounting  for  the  highest  variance  in  performance  (R^) 
were  employed  to  ascribe  scenario  difficulty.  These  difficulty 
predictor  weights  were  subsequently  used  to  assign  performance 
difficulty  levels  to  the  200  scenarios  developed  during  this 
effort  (See  Appendix  A). 


Scenario  Field  Testing 

Requirement :  To  subject  the  draft  scenarios,  scenario 

difficulty  indices,  task  and  summary  performance  measures,  SPM 
and  TPM  cut-off  values,  test  conditions,  and  scoring  system  to 
empirical  test  and  evaluation  on  representative  samples  of 
soldiers  and  weapons. 

Procedures:  Ten  Stinger  teams  from  the  Stinger  Platoon, 

Headquarters  and  Headquarters  Troop,  Third  Armored  Cavalry 
Regiment  stationed  at  Fort  Bliss,  Texas  participated  in  the 
study.  The  test  was  conducted  at  White  Sands  Missile  Range, 

White  Sands,  New  Mexico,  in  desert  terrain,  in  clear  weather, 
under  daylight  conditions,  during  the  month  of  January  of  1988, 
using  the  RADES  instrumented  testbed.  Each  Stinger  team  was 
exposed  to  as  many  as  twenty  scenarios.  Each  scenario  presented 
either  flying,  fixed-wing  (FW)  and/or  pop-up,  rotary-wing  (RW) 
targets,  either  in  single  or  multi-target  presentations. 

Scenario  specifications  are  provided  in  Table  17.  The  scenario 
sequences  executed  for  each  team  are  given  in  Table  18.  Each 
Stinger  team  (consisting  of  a  team  chief  and  gunner)  was  given  an 
early  warning  twenty  seconds  prior  to  each  scenario  presentation. 
Cuing  information  was  not  provided.  Team  chiefs  were  given 
binoculars,  which  they  used  for  aircraft  identification  purposes 
only. 

Task  performance  measures  (TPM)  included  times  and  ranges 
associated  with  target  detection,  interrogation,  acquisition, 
identification,  lock-on,  superelevation,  and  fire.  Summary  per¬ 
formance  measures  (SPM)  included  correctness  of  identification, 
engagement  effectiveness  (i.e.,  kill  or  miss),  hostile  attrition, 
hostile  ordnance  delivery  prevention,  and  fratricide.  Dependent 
variables  used  are  listed  and  described  in  Table  19. 


Taole  17 

Standardized  Scenario  Set 


SCEN 

NO. 

TYPE 

INTENT 

AIRCRAFT 

AZIMUTH 

ASPECT 

PATTERN/ 

RANGE 

PRESENT. 

ORDER 

DUR. 

LOD 

1 

FW 

F 

I  A10 

12 

0 

A 

MH 

2 

FW 

F 

F16 

1 

45 

B 

MH 

3 

FW 

F 

Fill 

10 

90 

C 

M 

4 

FW 

H 

MIG27 

12 

0 

A 

MH 

5 

FW 

H 

SU25 

11 

45 

B 

MH 

6 

FW 

H 

1  SU24 

2 

90 

C 

M 

7 

RW 

F 

UH60 

11 

90 

3 

25 

L 

8 

RW 

F 

CH3 

12 

45 

5 

25 

ML 

9 

RW 

H 

MI?  (HOKUM) 

11 

90 

3 

25 

L 

10 

RW 

H 

MI  8 

12/11 

45 

7/5 

SEQUEN 

10 

ML 

11 

RW 

H 

MI  24 

12/1 

0 

5/3 

SEQUEN 

10 

L 

12 

RW 

F 

AHl/UHl 

11/1 

45 

3 

SEQUEN 

25 

L 

13 

RW 

H 

MI8/MI24 

12/1 

45 

7/5 

SEQUEN 

25 

ML 

14 

RW 

F/H 

CH  3/MI  8 

11/1 

45 

5 

SIMULT 

40 

ML 

15 

RW 

H 

MI8/MI28/MI24 

11/12/1 

45 

3/5/5 

SIMULT 

50 

M 

16 

RW 

F/H/H 

AH1/MI24/MI28 

11/12/1 

90 

3/7/5 

SIMULT 

40 

M 

17 

RW 

F/H/F 

CH3/MI28/UH1 

11/12/1 

45 

5/5/3 

SIMULT 

40 

M 

18 

Both 

F/H 

A7/MI24 

12/1 

0/45 

A/3 

SIMULT  | 

30 

H 

19 

Both 

H 

SU17/MI7/MI28 

11/12/1 

45/45/0 

B/5/3 

SIMULT 

60 

H 

20 

Both 

H/F/H 

SU25/UH60/MI8 

12/11/1 

0/45/45 

A/5/5 

SIMULT 

40 

H 

Difficulty  levels:  H  =  high,  M=medium,  L  =  low 

FW  Patterns:  A  =  ingress  at  0  degrees  aspect;  3  =  diagonal  ingress  at  45 
degrees  aspect;  C  =  crossing  pattern  at  90  degrees  aspect 


Table  13 


Scenario 

Presentation  Sequence 

TEAMS 

SCENARIO  SEQUENCE 

1,2, 3, 4 

D,  9, 11, 13, 18, 10, 3, 8, 16, 2, 20, 5, 14, 7, 6, 15, 4, 17, 1,12 

5,6,7 

D, 15, 7, 14, 17, 4, 11, 5, 12, 20, 8, 6, 16, 18, 3, 9, 10, 1,13, 19, 2 

8,9,10 

D, 1,12, 10, 9, 2, 13 

KEY:  D  = 

Dummy  or  practice  trial 

oS 
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Dependent  Variables 


|  CODE 

TITLE  or  DESCRIPTION 

DUTY 

INTERPRETATION  | 

IDCOR 

Correctness  of 
Identification 

TL 

Number  of  correct 
ident i f icat i ons 
divided  by  number  of 
targets  identified 

FIRED 

Weapon  Fired 

G 

Number  of  weapon 
fires  divided  by 
number  of  targets 
presented . 

EFFECT 

Target  Hit  or  Missed 

-J 

Number  of  targets 
killed  divided  by 
number  engaged 

j  FRAT 

Fratricide 

BOTH 

EFFECT  on  Friendlies 

|  ATTRIT 

Attrition 

BOTH 

EFFECT  on  Hostiles  | 

ORD 

Ordnance  Prevention 

BOTH 

Number  of  hostiles 
delivering  ordnance 
divided  by  number 
presented 

RDET 

Range  of  Detection 

TL  or  G 

The  slant  range  from 
the  weapon  to  the 
target  when  the 
event  took  place; 
greater  ranges 
usually  indicate 
better  performance 
for  detection  and 
identification  but 
not  always  for  the 
other  events  (target 
can  be  inbound  or 
outbound).  Range 
is  relevant  for 
fixed-wing  targets 
only  since  rotary¬ 
wing  targets  simply 
pop-up  from  a  static 

RID 

Range  of  Identification 

TL 

RACQ 

Range  of  Initial 
Acquisition 

G 

RIFF 

Range  of  Interrogation 

G 

RLOCK 

Range  of  Lock-on 

G 

. 

1 RFIRE 

Range  of  Weapon  Fire 

G 

position.  Ranges 
are  in  full  scale 
kilometers. 


Taole  19  (Continued) 
Dependent  Variables 


j  CODE  j  TITLE  or  DESCRIPTION  j  DUTY  |  INTERPRETATION  j 

j  TDET 

j 

i 

Time  of  Detection 

TL  or  G 

Based  on  seconds 
after  target 
availability; 
availability  begins 
when  visual  line-of- 
sight  is  achieved  on 
the  first  RW  target 

TID 

Time  of  Identification 

TL 

Time  interval 
between  Detection 
and  Identification 

TACQ 

Time  of  Initial 
Acquisition 

G 

Time  interval 
between  Detection 
and  Acquisition 

TIFF 

1 

Time  of  Interrogation 

G 

Time  interval 
between  Detection 
and  IFF 

TLOCK 

Time  of  Lock-On 

G 

Time  interval 
between  Acquisition 
and  Lock-On 

TFIRE 

Time  of  Weapon  Fire 

G 

Time  interval 
between  Lock-On  and 
Fire 

TTOT 

Total  Engagement  Time 

BOTH 

Time  interval 
between  Detection 
and  Fire 

THAND 

Time  of  Handoff 

BOTH 

Time  interval 
between  Command  to 
Engage  and  Weapon 

F  ire 

KEY:  TL  =  Team  Leader;  G  =  Gunner 


Findings:  Table  Cl  (See  Appendix  C)  depicts  the  average 

performance,  number  of  cases,  and  the  variability  (standard 
deviation)  across  teams  for  each  scenario.  These  data  will  be 
used  as  a  benchmark  in  estimating  future  performance  on  this  same 
set  of  20  scenarios,  or  an  equivalent  sample  of  like-difficulty 
scenarios.  It  is  anticipated  that  future  performance  on  these 
scenarios  will  be  approximately  equal  for  soldiers  with  similar 
experience  and  ability,  and  that  performance  will  fall  within 
reasonable  boundaries  (90  percent  confidence  interval) 
established  in  this  table.  Data  contained  in  Table  Cl  are  self 
explanatory  and  therefore  require  no  detailed  elaboration  here. 

Tables  20  and  21  provide  the  summary  performance  of  soldiers 
according  to  scenario  conditions  across  all  observations.  As 
shown  in  Table  20,  soldiers  accurately  identified  friendly 
helicopters  45%  of  the  time  and  hostile  helicopters  71%  of  the 
time.  Friendly  FW  were  accurately  identified  69%  and  hostile  FW 
85%  of  the  time.  The  friendly  fixed  wing  F-16  and  the  friendly 
rotary  wing  CH-3  were  frequently  misident i f ied .  Attrition  rates 
on  hostile  aircraft  were  70%  for  FW  and  41%  for  RW. 

Further,  performance  effectiveness  (identification 
correctness:  IDCOR;  and  engagement  effect:  EFFECT)  decreased  in 

multiple  RW  target  scenarios  (See  Table  21).  Identification 
accuracy  and  engagement  effectiveness  (EFFECT,  ATTRIT,  and  ORD ) 
decreased  with  increased  workload.  Thus,  aircraft  model  type, 
intent,  and  the  number  of  targets  (workload  level),  were  deemed 
important  factors  for  assessing  scenario  difficulty  level. 

Table  22  shows  the  relationship  between  RW  elevation,  offset 
from  PTL ,  and  presentation  aspect  on  measures  of  engagement 
performance.  Data  are  summed  over  all  teams  and  targets  for  all 
relevant  scenarios.  Improved  target  visibility,  as  measured  by 
RW  target  elevation,  offset,  and  aspect,  resulted  in  improved 
performance  on  both  TPM  and  SPM.  Target  visibility  was  a  major 
factor  used  in  assessing  scenario  difficulty.  See  also  Table  Cl 
for  information  on  how  performance  varied  in  terms  of  target 
range,  aspect,  and  model  type. 

Tables  23  and  24  present  the  results  of  statistical  analyses 
which  show  that  observed  engagement  performance  varied  as  a 
function  of  scaled  scenario  difficulty  level.  Difficulty  level 
weights  were  assigned  to  scenarios  by  subject  matter  experts 
according  to  criteria  known  to  affect  engagement  performance, 
such  as  target  visibility,  workload,  intent,  range,  and  model 
type.  (See  Appendix  B).  Generally,  on  more  difficult  scenarios, 
troops  required  more  time  for  target  identification,  they  locked- 
onto  and  fired  at  identified  hostile  targets  later,  and  they 
required  more  time  for  a  complete  engagement  (detect  to  fire). 


Table  20 

Summary  Performance  Results  by  Target  Type  and  Intent 


VARIABLE 

TYPE 

INTENT 

MEAN 

SD 

N 

I  DOOR 

FW 

Fr i endly 

.6b 

.47 

26 

FW 

Hostile 

.85 

.36 

27 

FW 

Both 

.77 

.46 

53 

RW 

Friendly 

.45 

.  50 

62 

RW 

Hostile 

.71 

.46 

133 

RW 

Both 

.62 

.49 

195 

Both 

Both 

.66 

.48 

248 

PRAT 

FW 

Friendly 

.08 

.27 

26 

RW 

Friendly 

.  32 

.47 

62 

Bot  h 

Friendly 

.  25 

.44 

88 

ATTRIT 

FW 

Hostile 

.70 

.47 

27 

RW 

Hostile 

.41 

.49 

131 

Both 

Hostile 

.46 

.50 

158 

NOTE: 


Data  are  based  on  all  applicable  scenarios  and  teams. 
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Table  21 

Summary  Performance  Results  by  Each  Subsequent  RW 
Target  Worked 


VARIABLE 

TARGET 

MEAN 

SD 

N 

IDGOR 

1 

.72 

.45 

138 

2 

.62 

.49 

80 

3 

.43 

.  50 

30 

Ml 

.  66 

.48 

248 

EFFECT 

1 

.  77 

.43 

81 

2 

.58 

.50 

48 

3 

.  33 

.49 

14 

All 

.66 

.48 

144 

FRAT 

1 

.62 

.42 

59 

2 

.40 

.  50 

20 

3 

.11 

.  33 

9 

All 

.25 

.44 

88 

ATT R IT 

1 

.62 

.49 

79 

2 

.34 

.48 

58 

3 

.19 

.40 

21 

All 

.46 

.50 

158 

OxRD 

1 

.57 

.50 

79 

2 

.97 

.18 

60 

3 

.95 

.  22 

21 

All 

.77 

.42 

160 

NOTE:  Data  are  based  on  all  applicable  scenarios  and  teams 

(excluding  second  target  of  sequential  target  scenarios). 


Table  22 

Effects  of  RW  Scenario  Variations  on  Performance 
(Significant  Pearson  Correlations) 


INDEPENDENT 

VARIABLE 

DEPENDENT 

VARIABLE 

CORR.  | 

N 

PROB. 

RW  Elevation 

TDET 

-.33 

173 

.  000 

Above  Mask 

TID* 

-.34 

168 

.000 

( . 5  to  3.5 

TTOT 

-.19 

115 

.019 

degrees ; 

EFFECT 

-.15 

114 

.057 

mean  =  1.5) 

ORD 

-.28 

179 

.000 

RW  Offset 

TDET 

.18 

164 

.009 

From  PTL 

TID* 

.22 

160 

.003 

(1  to  55 

TID 

.16 

159 

.019 

degrees ; 

TACQ 

.  21 

133 

.007 

mean  =  15.2) 

TTOT 

.  19 

109 

.022 

RW  Aspect/ 

TDET 

-.13  | 

174 

.040 

Or ientat ion 

TID 

-.12 

171 

.067 

(0,  45  or  90 

TLOCK 

-.15 

116 

.052 

degrees ) 

IDCOR 

.12 

195 

.050 

ORD 

195 

.041 

*  =  Raw  TID;  time  from  availability  to  ID. 
N  =  Number  of  RW  target  presentat ions. 
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Table  23 

Results  of  t  Test  Comparisons  of  Performance  by  Difficulty  Level 
(Two-Tailed  Test  Using  Separate  Sample  Variances) 


OMPARI SON 

|  TYPE 

|  VARIABLE 

|  MEAN  ; 

SD  | 

N  j 

T  i 

DF  | 

PROB 

H 

V  3 

M 

FW 

TID 

21.9; 13.0 

1 1 . 7  ;  3 . 6 

16;  14 

2.9 

18.2 

.009 

H 

VS 

M 

FW 

RLOCK 

3.7;  5.7 

1 . 7  ;  0 . 5 

10;  9 

3.5 

10.4 

.006 

H 

VS 

M 

FW 

RFIRE 

3.0;  5.3 

1 . 4  ;  0 . 6 

11;  8 

5.9 

13.7 

.000 

H 

vs 

.-1 

FW 

TTOT 

29 . 2 ; 20 . 5 

12 . 9 ; 3 . 8 

11;  8 

2.1 

12.3 

.056 

MH 

VS 

M 

FW 

TID 

17. 2; 13.0 

6 . 5  ;  3 . 6 

20;  14 

2.4 

30.8 

.024 

MH 

vs 

M 

F  W 

RLOCK 

4.3;  5.7 

1 . 9  ;  0 . 5 

10;  9 

2.1 

10.2 

.058 

MH 

vs 

M 

FW 

RFIRE 

3.8;  5.8 

2 . 1 ;  0 . 6 

10;  8 

2.8 

10.6 

.017 

MH 

vs 

M 

FW 

TTOT 

25 . 3 ; 20 . 5 

7 . 7  ;  3 . 8 

10;  8 

1.7 

13.6 

.  108 

M 

vs 

L 

RW 

TTOT 

11.1;  8.9 

3 . 8  ;  3 . 8 

37  ;  20 

2.1 

39.0 

.046 

MH 

vs 

M 

3ot  h 

TID 

17.2;  7.1 

6 . 5  ;  4 . 4 

20 ;  70 

6.5 

24.3 

.  000 

MH 

vs 

M 

Bot  h 

TTOT 

25 . 3 ; 11 . 8 

7 . 7  ;  5 . 8 

10;  49 

5.2 

11.1 

.000 

MH 

vs 

ML 

Both 

TIu 

17.2;  5.6 

6. 5; 2 . 6 

20 ;  49 

7.7 

21.6 

.000 

MH 

vs 

ML 

Both 

TTOT 

25. 3;  11.  i 

7 . 7  ;  3 . 8 

10 ;  37 

5.7 

10.2 

.000 

H 

vs 

M 

Both 

TID 

12.8;  7.1 

11 . 5 ; 4 . 4 

37  ;  70 

2.9 

41.7 

.006 

H 

vs 

M 

Bot  h 

TFIRE 

4.5;  2.5 

5 . 3  ;  1 . 9 

29;  43 

2.0 

32.9 

.054 

H 

vs 

H  1 

Both 

TTOT 

19.9;11.8 

12.8; 5.8 

29;  49 

3 . 2  j 

34.8  , 

.003 

N  -  Number  of  applicable  FW  or  RW  target  presentations. 


Table  24 

Relationship  Between  Difficulty  Level  and  Performance 
(Significant  Kendall  and  Spearman  Correlations) 


DEPENDENT 

KENDALL ' S 

SPEARMAN ' S 

TYPE 

VARIABLE 

TAU 

N 

PROB 

RHO 

N 

PROB 

FW 

TID 

.85 

9 

.002 

.94 

9 

.000 

FW 

RFIRE 

-.91 

4 

.035 

-.95 

4 

.026 

TTOT 

-.54 

6 

.075 

-.74 

6 

.046 

FRAT 

.80 

5 

.039 

.88 

5 

.023 

N  =  Number  of  Stinger  teams  for  which  applicable  data  were 
available  . 
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There  were  exceptions  to  the  above.  For  example,  a  scenario 
with  a  crossing  FW  pattern  was  determined  to  be  easier  than  one 
with  an  ingressing  FW  pattern  because  the  target  was  easier  to 
detect  and  identify,  and  was  more  frequently  identified 
correctly,  However,  it  was  also  available  for  a  shorter  period 
of  time,  forcing  the  soldiers  to  complete  the  engagement  sooner. 
Further,  although  more  workload  implied  more  difficulty, 
sometimes  it  resulted  in  shorter  engagement  times  since  the 
soldiers  were  rushed.  Therefore,  both  an  easy  and  a  difficult 
scenario  was  character ized  by  shorter  engagement  times,  depending 
upon  the  conditions.  Increasing  the  number  of  targets  often 
resulted  both  in  higher  hostile  attrition  rates  and  higher 
fratricide  rates,  since  the  soldiers  were  inclined  to  engage 
everything  that  appeared  to  pose  a  threat.  As  a  rule,  however, 
increases  in  difficulty  resulted  both  in  decreased  engagement 
efficiency  and  decreased  engagement  effectiveness. 

scenario  difficulty  in  this  study,  as  measured  by 
performance  effectiveness  and  efficiency,  was  primarily 
attributed  to: 


•  Target  characteristics 

-  Model  Size 

-  Intent 

-  Type  (RW  versus  FW) 


•  Visibility  conditions 

-  Target  Elevation 

-  Target  Offset 

-  Target  Aspect 

-  Target  Speed 


•  Workload  level 

-  Number  of  Targets 


This  test  empirically  validated  many  of  the  difficulty 
factors  used  by  the  experts  to  ascribe  weights  to  scenarios. 
Studies  such  as  this  enable  researchers  to  estimate  the 
difficulty  of  air  defense  scenarios  from  empirical  evidence.  The 
difficulty  scaling  technique  used  to  assess  the  difficulty  level 
of  the  200  scenarios  presented  in  Appendix  A  was  based  upon  this 
empirical  evidence. 


If 


Determination  of  SPM  and  TPM  Cut-off  Scores 


Requirement^  To  establish  SPM  score  cut-off  values  which 
sort  3H0RAD  crews,  teams,  and  operators  into  qualified  and 
unqualified  groups.  It  was  required  that  S^M  cut-off  values  be 
achievable,  valid,  reliable,  practical,  and  economic  to 
administer.  A  second  requirement  was  to  establish  TPM  score  cut¬ 
off  values  which  identified  deficient  part-task  performance  and 
which  indicated  the  sources  of  failure  to  qualify. 


Procedures :  It  was  acknowledged  from  the  onset  of  the 

present  research  that  only  the  USAADASCH,  Directorate  of  Training 
and  Doctrine  ( DOTD ) ,  Fort  Bliss,  is  chartered  to,  and  proponent 
for,  setting  range  qualification  standards.  Therefore  the 
present  researcn  was  limited  to  analysis,  interpretation,  and 
recommendation  of  SPM  and  TPM  cut-off  scores. 

Analysts  examined  the  results  of  the  RADSS  meta-analysis, 
field  test  data,  literature  on  threat  and  friendly  air 
operations,  airspace  management,  command,  control  and 
communications,  and  weapons  capabilities  and  limitations.  Then 
analysts  assisted  ARI  in  establishing  realistic  SPM  cut-off 
scores,  and  in  adapting  the  draft  training  and  qualification 
scenarios  to  the  current  tactical  Short  Range  Air  Defense 
picture.  It  must  be  noted  that  data  from  the  Vulcan  or  PIVADS 
weapons  were  not  available.  It  was  therefore  assumed  that  the 
criteria  would  need  to  be  adjusted  to  accomodate  close-range 
weapons  such  as  these. 

Draft  cut-off  scores  were  subjected  to  empirical  field 
testing  using  the  RAD  SS  testbed.  The  field  tests  were  conducted 
to  insure  that  the  current  3H0RAD  soldier  population  could  attain 
the  performance  standards,  and  that  those  standards  could  be 
reliably,  practically,  and  economically  administered.  The 
Stinger  test  described  earlier  helped  to  serve  this  purpose,  and 
verified  that  these  requirements  were  met. 


bindings:  Table  25  provides  the  recommended  SPM  cut-off 

scores  established  by  the  present  program  of  research.  Table  26 
provides  the  recommended  TPM  cut-off  scores  which  are  to  be  used 
in  diagnosing  the  sources  of  crew,  team,  and  operator  failures  to 
qualify  on  SPM.  It  should  be  noted  that  all  TPM  ranges  reflect 
incoming  (not  outgoing)  FW  targets. 


Table  25 

3PM  Cutoff  Values  Estimated  From  RADES  Research  Results 


FIXED  WING 

ROTAR7  WING 

1 

SPM 

LOD 

VALUE  J 

SPM 

LOD 

VALUE  j 

Identity 

H 

70% 

Identity 

H 

70% 

Correctness 

M 

75% 

Correctness 

M 

75% 

( IDCOR ) 

L 

80% 

(IDCOR) 

L 

80% 

Friends 

H 

70% 

Friends 

H 

70% 

Identified 

M 

75% 

Identified 

M 

75% 

( FIDCOR ) 

80% 

( FIDCOR ) 

L 

80% 

H os tiles 

I  I 

75% 

Hostiles 

H 

75% 

Identi f ied 

M 

80% 

Identified 

M 

80% 

(HIDCOR) 

L 

85% 

(HIDCOR ) 

L 

85% 

Friends 

H 

30% 

Friends 

H 

30% 

Engaged 

M 

25% 

Engaged 

M 

25% 

(  FENG  ) 

L 

20% 

( FENG  ) 

L 

20% 

Hostiles 

H 

75% 

Hostiles 

H 

75% 

Engaged 

M 

80% 

Engaged 

M 

80% 

( HENG  ) 

L 

85% 

(HENG  ) 

L 

85% 

Fr iendl y 

H 

25% 

Friendly 

H 

25% 

Kills 

M 

20% 

Kills 

M 

20% 

( FRAT ) 

L 

15% 

(FRAT) 

L 

15% 

Hostile 

H 

45% 

Host ile 

H 

55% 

Kills 

M 

60% 

Kills 

M 

70% 

( ATTRIT) 

L 

75% 

(ATTRIT) 

L 

80% 

Engaged/ 

H 

60% 

Engaged/ 

H 

75% 

Destroyed 

M 

75% 

Destroyed 

M 

85% 

(EFFECT) 

90% 

(EFFECT) 

L 

95% 

Ordnance 

1 

95% 

Ordnance 

H 

95% 

Released 

M 

75% 

Released 

M 

75% 

(ORD  ) 

L 

30% 

(ORD ) 

L 

30% 

jOD  =  Level  of  Difficulty 


IS 
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TPM  Cutoff  Values  Estimated  From  RADES  Research  Results 


|  FIXED  WING 

1 

ROTARY  WING 

1 

j  TPM 

LOD 

VALUE  j 

TPM 

LOD 

VALUE  | 

Detect 

H 

8.0 

km 

Detect 

H 

10.0 

sec 

( RDET  ) 

M 

11.0 

km 

( TDET ) 

M 

6.0 

sec 

L 

14.0 

km 

L 

4.0 

sec 

Acquire 

H 

5.0 

km 

Acquire 

H 

6.0 

sec 

(RACQ) 

M 

6.0 

km 

(TACQ) 

M 

5.0 

sec 

L 

7.0 

km 

L 

4.0 

sec 

Identify 

H 

4.0 

km 

Identify 

H 

9.0 

sec 

(RID) 

M 

6.0 

km 

(TID  ) 

M 

7.0 

sec 

L 

8.0 

km 

L 

5.0 

sec 

Engage 

H 

3.5 

km 

Engage 

H 

2.0 

sec 

( RENG  ) 

M 

5.5 

km 

(TENG  ) 

M 

1.0 

sec 

L 

7.5 

km 

L 

1.0 

sec 

Lock-On 

H 

4.0 

km 

Lock 

H 

6.0 

sec 

( RLOCK ) 

M 

5.0 

km 

( TLOCK ) 

M 

4.0 

sec 

6.0 

km 

L 

2.0 

sec 

Fire 

I  1 

2.0 

km 

Fire 

H 

5.0 

sec 

(RFIRE ) 

M 

4.0 

km 

(TFIRE ) 

M 

3.0 

sec 

L 

5.0 

km 

L 

2.0 

sec 

LOD  =  Level 

of  Difficulty 

Total 

H 

15.0 

sec 

( TTOT ) 

M 

12.0 

sec 

L 

8.0 

sec 

It  can  be  seen,  when  comparing  the  Stinger  test  results 
cited  earlier  (Table  Cl  and  Table  20)  to  the  criteria  established 
a-priori  (Tables  25  and  26)  that  soldiers  met  some  of  the 
criteria  but  failed  to  meet  others.  For  example,  while 
correctness  of  target  identification  was  often  within  tolerance 
limits,  ordnance  prevention  and  fratricide  rates  often  were  not. 
This  is  consistent  with  the  observation  made  earlier  that  many  of 
the  system  and  operator  performance  standards  were  not  currently 
being  achieved.  This  may  be  due  in  part  to  the  fact  that  the  TPM 
and  3PM  estimates  were  based  on  cued  trials,  and  the  Stinger  test 
results  were  based  on  non-cued  trials.  The  decision  to  base  the 
criteria  on  cued  trials  was  reached  after  the  Stinger  test  was 
conducted.  Overall,  however,  the  Stinger  test  results  cited 
earlier  were  consistent  with  the  TPM  and  SPM  estimates.  For 
example,  event  ranges  for  moderately  difficult  FW  scenarios  for 
ietection,  identification,  acquire,  and  fire  were  estimated  at 
about  11,  6,  6,  and  4  kilometers,  respectively.  These  estimated 
ranges  were  similar  to  those  obtained  in  the  Stinger  test. 
Further,  these  events  for  low  difficulty  RW  scenarios  were 
estimated  at  about  4,  5,  4,  and  2  seconds,  respectively.  Again 
these  estimated  values  were  consistent  with  performance  in  the 
Stinger  test. 

SPM  and  TPM  cut-off  scores  were  determined  as  a  function  of 
scenario  difficulty.  Definition  of  meaningful  SPM  cutoff  levels 
was  accomplished  as  a  natural  by-product  of  the  meta-analysis, 
and  the  RADES  experiment  described  earlier.  TPM  cut-off  scores 
were  established  in  the  same  way.  It  is  currently  anticipated, 
however,  that  TPM  cut-off  scores  will  be  used  solely  for 
diagnosing  the  sources  of  SPM  failures  to  qualify,  and  will  not 
be  the  basis  for  crew,  team,  or  operator  pass  or  fail 
determinations. 
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APPENDIX  A 


RTS  SCENARIO  LIBRARY  AND  TARGET  SPECIFICATIONS 


Seen 

Mo. 

No. 

Targ 

|  Type 

Intent 

Aircraft 

Model 

Clock 

Azimuth 

Degrees 

Aspect 

Pattern 

/Stand 

Pres.  : 
Order 

Seconds  1 
Avail . j 

Dif f ic 
Level 

1 

1 

FW 

F 

1 

10 

90 

A 

— — 

M 

2 

1 

FW 

F 

1 

11 

45 

B 

— 

— 

MH 

3 

1 

FW 

F 

1 

12 

0 

C 

— 

— 

MH 

4 

1 

FW 

F 

1 

1 

45 

D 

— 

— 

MH 

5 

1 

FW 

F 

1 

2 

90 

E 

— 

— 

M 

6 

1 

FW 

F 

2 

10 

90 

A 

— 

— 

M 

7 

1 

FW 

F 

2 

11 

45 

B 

— 

— 

MH 

a 

1 

FW 

F 

2 

12 

0 

C 

— 

— 

MH 

9 

1 

FW 

F 

2 

1 

45 

D 

— 

— 

MH 

10 

1 

FW 

F 

2 

2 

90 

E 

— 

— 

M 

n 

1 

FW 

F 

3 

10 

90 

A 

— 

— 

M 

12 

1 

FW 

F 

3 

11 

45 

B 

— 

— 

MH 

13 

1 

FW 

F 

3 

12 

0 

C 

— 

— 

MH 

14 

1 

FW 

F 

3 

1 

45 

D 

— 

— 

MH 

15 

1 

FW 

F 

3 

2 

90 

E 

— 

— 

M 

16 

1 

FW 

H 

4 

10 

90 

A 

— 

— 

M 

17 

1 

FW 

H 

4 

11 

45 

B 

— 

— 

MH 

18 

1 

FW 

H 

4 

12 

0 

C 

— 

— 

MH 

19 

1 

FW 

H 

4 

1 

45 

D 

— 

— 

MH 

20 

1 

FW 

H 

4 

2 

90 

t? 

— 

— 

M 

21 

1 

FW 

H 

5 

10 

90 

A 

— 

— 

M 

22 

1 

FW 

H 

5 

11 

45 

B 

— 

— 

MH 

23 

1 

FW 

H 

5 

12 

0 

C 

— 

— 

MH 

24 

1 

FW 

H 

5 

1 

45 

D 

— 

— 

MH 

25 

1 

FW 

H 

5 

2 

90 

E 

— 

— 

M 

26 

1 

FW 

H 

6 

10 

90 

A 

— 

— 

M 

27 

1 

FW 

H 

6 

11 

45 

B 

— 

— 

MH 

28 

1 

FW 

H 

6 

12 

0 

C 

— 

— 

MH 

29 

1 

FW 

H 

6 

1 

45 

D 

— 

— 

MH 

30 

1 

FW 

H 

6 

2 

90 

E 

— 

— 

M 

31 

2 

FW 

F 

1*2 

11 

45 

B 

Sequen 

30 

H 

32 

2 

FW 

F 

2,3 

11,12 

45,0 

D,C 

Sequen 

30 

H 

33 

2 

FW 

F 

3 

12 

0 

C 

Sequen 

30 

H 

34 

2 

FW 

H 

4,5 

1 

45 

B 

Sequen 

30 

H 

35 

2 

FW 

H 

5,6 

11,12 

45,0 

D,C 

Sequen 

30 

H 

36 

2 

FW 

H 

6 

12 

0 

C 

Sequen 

30 

H 

37 

2 

FW 

F,H 

3,6 

11,1 

45 

B,D 

Sequen 

30 

H 

38 

2 

FW 

H,F 

6,3 

1,11 

45 

D,B 

Sequen 

30 

H 

39 

2 

FW 

F ,  H 

2,4 

12 

0 

C 

Sequen 

30 

H 

40 

2 

FW 

H  ,  F 

4,2 

1 

90 

E 

Sequen 

30 

MH 

41 

2 

FW 

F 

1 

12 

0 

C 

Simult 

— 

MH 

42 

2 

FW 

H 

5 

12 

0 

C 

Sirault 

— 

MH 

43 

2 

FW 

F 

1,3 

12,2 

0,90 

C,  E 

Simult 

— 

XH 

44 

2 

FW 

F 

2 

10,12 

90,0 

A ,  C 

Simult 

— 

XH 

45 

2 

FW 

H 

4,6 

12,2 

0,90 

C,  E 

Simult 

— 

XH 

A1 


Seen 
No . 

NO. 

Targ 

Type 

Intent 

Aircraft 

Model 

Clock 

Azimuth 

Degrees 

Aspect 

Pattern 

/Stand 

Pres. 

Order 

Seconds 
Avail . 

Dif f ic 
Level 

46 

2 

FW 

H 

4 

10,12 

90,0 

A,  C 

Simult 

XH 

47 

2 

FW 

F,H 

2,5 

11,1 

45 

3 ,  D 

Simult 

— 

XH 

48 

2 

FW 

H ,  F 

6,1 

10,2 

90 

A,  E 

S imult 

— 

XH 

49 

2 

FW 

F/  H 

2,5 

10,11 

90,45 

A, 3 

Simult 

— 

XH 

50 

2 

FW 

H  ,  F 

6,1 

1,2 

45,90 

D,E 

Simult 

— 

XH 

51 

1 

RW 

F 

11 

11 

45 

1 

— 

25 

L 

52 

1 

RW 

F 

11 

12 

90 

4 

— 

25 

L 

53 

1 

RW 

F 

11 

1 

0 

5 

— 

25 

L 

54 

1 

RW 

F 

11 

1 

45 

6 

— 

25 

ML 

55 

1 

RW 

F 

12 

11 

0 

1 

— 

25 

L 

56 

1 

RW 

F 

12 

11 

90 

1 

— 

25 

L 

57 

1 

RW 

F 

12 

12 

45 

3 

— 

25 

L 

58 

1 

RW 

F 

12 

1 

45 

6 

— 

25 

ML 

59 

1 

RW 

F 

13 

11 

45 

2 

— 

25 

ML 

60 

1 

RW 

F 

13 

12 

45 

4 

— 

25 

ML 

61 

1 

RW 

F 

13 

1 

0 

5 

— 

25 

L 

62 

1 

RW 

F 

13 

11 

90 

6 

— 

25 

L 

63 

1 

RW 

F 

14 

11 

0 

2 

— 

25 

ML 

64 

1 

RW 

F 

14 

12 

45 

3 

— 

25 

L 

65 

1 

RW 

F 

14 

12 

90 

4 

— 

25 

ML 

66 

1 

RW 

F 

14 

1 

45 

5 

— 

25 

L 

67 

1 

RW 

H 

15 

11 

45 

1 

— 

25 

L 

63 

1 

RW 

H 

15 

11 

45 

2 

— 

25 

ML 

69 

1 

RW 

H 

15 

12 

0 

3 

— 

25 

L 

70 

1 

RW 

H 

15 

1 

90 

6 

— 

25 

L 

71 

1 

RW 

H 

16 

11 

0 

1 

— 

25 

L 

72 

1 

RW 

H 

16 

12 

45 

3 

— 

25 

L 

73 

1 

RW 

H 

16 

12 

90 

4 

— 

25 

L 

74 

1 

RW 

H 

16 

1 

45 

6 

-- 

25 

ML 

75 

1 

RW 

H 

17 

11 

45 

1 

— 

25 

L 

76 

1 

RW 

H 

17 

12 

0 

3 

— 

25 

L 

77 

1 

RW 

H 

17 

1 

45 

5 

— 

25 

L 

78 

1 

RW 

H 

17 

1 

90 

6 

— 

25 

L 

79 

1 

RW 

H 

18 

11 

90 

2 

— 

25 

L 

80 

1 

RW 

H 

18 

11 

45 

2 

— 

25 

ML 

81 

1 

RW 

H 

18 

12 

45 

4 

— 

25 

ML 

82 

1 

RW 

H 

18 

1 

0 

5 

— 

25 

L 

83 

2 

RW 

F 

11 

12 

0 

4,2 

Sequen 

15 

L 

84 

2 

RW 

F 

13 

12,1 

90 

4,6 

Sequen 

15 

L 

85 

2 

RW 

F 

14 

11,12 

45 

1,4 

Sequen 

15 

L 

86 

2 

RW 

F 

13,12 

1,11 

45 

6,1 

Sequen 

15 

L 

87 

2 

RW 

H 

15 

11 

0 

1,2 

Sequen 

15 

L 

88 

2 

RW 

H 

16 

12,1 

90 

4,6 

Sequen 

15 

L 

89 

2 

RW 

H 

17 

1,12 

45 

5,4 

Sequen 

15 

L 

90 

2 

RW 

H 

15,18 

11,1 

45 

1,6 

Sequen 

15 

L 

91 

2 

RW 

F,H 

12,18 

11,1 

0 

1,5 

Sequen 

15 

L 

92 

2 

RW 

H »  F 

18,12 

11,1 

0 

2,6 

Sequen 

15 

ML 

93 

2 

RW 

F,H 

11,17 

12 

0 

4,3 

Sequen 

15 

ML 

A  2 


Seen 

MO. 

Aircraft 

Clock 

Degrees 

Pattern 

Pres. 

Seconds 

Dif f ic 

Mo. 

Targ 

Type 

Intent 

Model 

Azimuth 

Aspect 

/Stand 

Order 

Avail . 

Level 

94 

2 

RW 

H,F 

16,14 

12 

45 

4,3 

Sequen 

15 

L 

95 

2 

RW 

F,H 

13,15 

11,1 

90 

2,6 

Sequen 

15 

L 

96 

2 

RW 

H ,  F 

12,16 

11,1 

0,45 

1,6 

Sequen 

15 

L 

97 

2 

RW 

F 

11 

11,1 

0 

1,5 

Simult 

40 

ML 

98 

2 

RW 

F 

12,13 

11,1 

90 

2,6 

Simult 

40 

ML 

99 

2 

RW 

F 

11,14 

12 

45 

4,3 

Simult 

40 

ML 

100 

2 

RW 

F 

12,13 

11,12 

0,45 

1,4 

Simult 

40 

ML 

101 

2 

RW 

H 

17 

11,1 

0 

1,5 

Simult 

40 

ML 

102 

2 

RW 

H 

18,17 

11,1 

90 

2,6 

Simult 

40 

ML 

103 

2 

RW 

H 

16,15 

12 

45 

4,3 

Simult 

40 

ML 

104 

2 

RW 

H 

18,15 

11,12 

0,45 

1,4 

Simult 

40 

ML 

105 

2 

RW 

F,  H 

12,18 

11,1 

0 

1,5 

Simult 

40 

ML 

106 

2 

RW 

F,H 

11,16 

11,12 

45 

4,6 

Simult 

40 

ML 

107 

2 

RW 

F  /  H 

13,17 

11,1 

45,0 

2,5 

S  imult 

40 

ML 

108 

2 

RW 

F,H 

12,18 

12 

90 

4,2 

Simult 

40 

ML 

109 

2 

RW 

H  ,  F 

15,14 

1 

45 

6,5 

Simult 

40 

ML 

110 

2 

RW 

H,F 

15,13 

11,1 

45 

2,6 

Simult 

40 

ML 

111 

2 

RW 

H,F 

17,11 

11,12 

0,45 

1,4 

Simult 

40 

ML 

112 

2 

RW 

H,F 

16,12 

12,1 

90,45 

4,6 

Simult 

40 

ML 

113 

3 

RW 

F 

11,14,13 

12,1,11 

45 

4,5,2 

Sequen 

15 

ML 

114 

3 

RW 

F 

13,14,12 

11,12,11 

45 

2,4,1 

Sequen 

15 

ML 

115 

3 

RW 

H 

18,1.5,17 

12,11,1 

45 

4,2,5 

Sequen 

15 

ML 

116 

3 

RW 

H 

16,18,17 

1,12,1 

45 

6,4,5 

Sequen 

15 

ML 

117 

3 

RW 

F  ,  H  ,  F 

13,16,12 

11,1,12 

45 

2,5,3 

Sequen 

15 

ML 

118 

3 

RW 

F,H,F 

11,18,12 

11,12,1 

90 

1,4,6 

Sequen 

15 

ML 

119 

3 

RW 

F,H,F 

14,16,13 

12,11,1 

0 

4,1,5 

Sequen 

15 

M 

120 

3 

RW 

f  #  f  *  a 

13,12,15 

12,11,11 

0,45,45 

4,1,2 

Sequen 

15 

ML 

121 

3 

RW 

H,F,F 

16,11,11 

12,1,1 

45,45,0 

4,6,5 

Sequen 

15 

ML 

122 

3 

RW 

h,f,h 

17,12,15 

1,12,11 

45 

6,3,2 

Sequen 

15 

ML 

123 

3 

RW 

H  ,  F  ,  H 

18,11,17 

11,12,1 

90 

2,4,5 

Sequen 

15 

ML 

124 

3 

RW 

H  ,  F ,  H 

18,14,15 

1,12,11 

0 

5,4,2 

Sequen 

15 

M 

125 

3 

RW 

F,H,H 

14,18,17 

12,11,11 

0,45,45 

4,1,2 

Sequen 

15 

ML 

126 

3 

RW 

H,H.F 

16,17,13 

12,1,1 

45,45,0 

4,6,5 

Sequen 

15 

ML 

127 

3 

RW 

F 

13,14,11 

11,12,1 

45 

2,4,6 

Simult 

60 

M 

128 

3 

RW 

F 

12,14,13 

11,12,1 

45 

1,3,5 

Simult 

60 

M 

129 

3 

RW 

F 

12,11,12 

11,12,1 

0,90,45 

1,4,6 

Simult 

60 

M 

130 

3 

RW 

F 

14,11,13 

11,1,1 

90,45 

2,5,6 

Simult 

60 

M 

131 

3 

RW 

H 

15,16,17 

11,12,1 

45 

2,4,6 

Simult 

60 

M 

132 

3 

RW 

H 

17,15,18 

11,12,1 

45 

1,3,5 

Simult 

60 

M 

133 

3 

RW 

H 

16,18,16 

11,12,1 

0,90,45 

1,4,6 

Simult 

60 

M 

134 

3 

RW 

H 

18,17,16 

11,1,1 

90,45 

2,5,6 

Simult 

60 

M 

135 

3 

RW 

F,H,F 

11,17,11 

11,12,1 

0 

1,3,5 

Simult 

60 

M 

136 

3 

RW 

F,H,F 

12,16,13 

1,12,11 

45 

6,4,2 

Simult 

60 

M 

137 

3 

RW 

F,H,F 

14,15,13 

12,1,11 

90 

4,6,2 

Simult 

60 

M 

138 

3 

RW 

F,F,H 

14,13,17 

11,12,1 

0,90,45 

2,4,5 

Simult 

60 

M 

139 

3 

RW 

F,F,H 

12,11,15 

11,12,1 

0,45,0 

1,4,6 

Simult 

60 

M 

140 

3 

RW 

H,F,F 

15,12,13 

11,12,1 

0,90,45 

2,3,6 

Simult 

60 

M 

141 

3 

RW 

H ,  F,  F 

16,14,11 

11,12,1 

45,0,90 

1,3,6 

Simult 

60 

M 
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142 

3 

RW 

F,F,H 

11,13,15 

11,1,11 

45 

1,5,2 

Simult 

60 

M 

143 

3 

RW 

H  ,  F  ,  F 

17,14,12 

1,12,1 

45 

5,4,6 

Simult 

60 

M 

144 

3 

RW 

8,F,8 

17,12,17 

11,12,1 

0 

1,3,5 

Simult 

60 

M 

145 

3 

RW 

H,F,8 

16,11,15 

1,12,11 

45 

6,4,2 

Simult 

60 

M 

146 

3 

RW 

H,F,H 

16,11,15 

12,1,11 

90 

4,6,2 

Simult 

60 

M 

147 

3 

RW 

rt,H,F 

18,16,12 

11,12,1 

0,90,45 

2,4,5 

Simult 

60 

M 

148 

3 

RW 

h,h,f 

15,16,13 

11,12,1 

0,45,0 

1,4,6 

Simult 

60 

M 

149 

3 

RW 

F ,  8  ,  H 

14,17,15 

11,12,1 

0,90,45 

2,3,6 

Simult 

60 

M 

150 

3 

RW 

F,H,H 

12,15,17 

11,12,1 

45,0,90 

1,3,6 

Simult 

60 

M 

151 

3 

RW 

H  »  H  *  F 

17,18,13 

11,1,11 

45 

1,5,2 

S  iraul  t 

60 

M 

152 

3 

RW 

F,8,B 

11,18,16 

1,12,1 

45 

5,4,6 

Simult 

30 

M 

153 

2 

MIX 

F 

1,13 

12 

0,45 

C ,  4 

Simult 

40 

H 

154 

2 

MIX 

F 

2,11 

1,11 

45 

D,1 

Simult 

40 

H 

155 

2 

MIX 

F 

3,12 

12,1 

0,90 

C,  6 

Simult 

40 

H 

156 

2 

MIX 

F 

2,14 

10,1 

90,45 

A, 5 

Simult 

40 

H 

157 

2 

MIX 

F 

1,14 

12 

0 

C,3 

S imult 

40 

H 

158 

2 

MIX 

F 

3,11 

1 

45 

D ,  6 

Simult 

40 

H 

159 

•> 

MIX 

8 

4,18 

12 

0,45 

C,4 

Simult 

40 

8 

160 

2 

MIX 

H 

5,17 

11,1 

45 

B,  5 

Simult 

40 

a 

161 

2 

MIX 

H 

6,15 

12,1 

0,90 

C,6 

Simult 

40 

H 

162 

2 

MIX 

H 

5,16 

2,11 

90,45 

E,1 

Simult 

40 

H 

163 

2 

MIX 

H 

6,17 

12 

0 

C ,  3 

Simult 

40 

8 

164 

2 

MIX 

8 

4,15 

11 

45 

8,2 

Simult 

40 

8 

165 

2 

MIX 

F,  H 

2,18 

12 

0,45 

C,4 

Simult 

40 

H 

166 

2 

MIX 

F,S 

3,16 

11,1 

45 

B ,  6 

Simult 

40 

H 

167 

2 

MIX 

F,B 

2,16 

12,1 

0,90 

C,  6 

Simult 

40 

H 

168 

2 

MIX 

F,H 

1,17 

2,11 

90,45 

E,1 

Simult 

40 

H 

169 

2 

MIX 

F  ,  H 

3,15 

12 

0 

C,  3 

Simult 

40 

H 

170 

2 

MIX 

F,H 

1,15 

11 

45 

3,2 

Simult 

40 

8 

171 

2 

MIX 

H,F 

5,11 

12 

0,45 

C,  4 

Simult 

40 

8 

172 

2 

MIX 

H,F 

6,13 

1,11 

45 

D,2 

Simult 

40 

H 

173 

2 

MIX 

B,F 

4,14 

12,11 

0,90 

C,2 

Simult 

40 

8 

174 

2 

MIX 

8,F 

4,11 

10,1 

90,45 

A, 5 

Simult 

40 

8 

175 

2 

MIX 

H  /  F 

5,12 

12 

0 

C,  3 

Simult 

40 

8 

176 

2 

MIX 

H,F 

6,12 

1 

45 

D ,  6 

Simult 

40 

8 

177 

3 

MIX 

F 

1,12,11 

12,11,1 

0,45,45 

C,  1 , 5 

Simult 

60 

8 

178 

3 

MIX 

F 

2,14,14 

12 

0 

C ,  3 , 4 

Simult 

60 

8 

179 

3 

MIX 

F 

3,11,13 

1,1,11 

45 

D ,  6 , 2 

Simult 

60 

H 

180 

3 

MIX 

F 

2,14,12 

10,12,1 

90,0,45 

A, 3, 6 

Simult 

60 

8 

181 

3 

MIX 

8 

4,16,18 

12,11,1 

0,45,45 

C,  1, 5 

Simult 

60 

8 

182 

3 

MIX 

H 

6,17,17 

12 

0 

C,  3 ,4 

Simult 

60 

8 

133 

3 

MIX 

8 

5,15,16 

11,11,1 

45 

B,  2 , 6 

Simult 

60 

8 

184 

3 

MIX 

8 

4,16,15 

10,1,12 

90,45,0 

A, 6, 3 

Simult 

60 

8 

185 

3 

MIX 

F,F,8 

1,14,16 

12,11,1 

0,45,45 

C ,  2 , 6 

S imult 

60 

8 

186 

3 

MIX 

F,F,H 

2,12,18 

11,11,1 

45,0,0 

B  ,  1 , 5 

Simult 

60 

H 

187 

3 

MIX 

F,H,F 

3,17,13 

12 

0 

C,  3 , 4 

S imult 

60 

H 

188 

3 

MIX 

F,H,F 

1,15,11 

1,12,1 

45 

D ,  2 , 6 

S imult 

60 

8 
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189 

3 

MIX 

F ,  H  »  H 

2,16,18 

2,1,12 

90,45,0 

E ,  6 , 4 

Sirault 

60 

190 

3 

MIX 

F,H,H 

3,17,17 

12,11,1 

45,45,0 

C,  1 , 5 

Simult 

60 

191 

3 

MIX 

F,H,H 

2,18,15 

10,11,12 

90,45,45 

A, 2, 3 

Sirault 

60 

192 

3 

MIX 

H  »  H  ,  F 

4,15,12 

12,11,1 

0,45,45 

C,  2 , 6 

Sirault 

60 

193 

3 

MIX 

H,H,F 

5,16,11 

11,11,1 

45,0,0 

B ,  1 , 5 

Simult 

60 

194 

3 

MIX 

H,H,F 

5,15,11 

11,11,1 

45,0,0 

B  ,  2 , 6 

Simult 

60 

195 

3 

MIX 

H  , F,  H 

6,14,18 

12 

0 

C ,  3 , 4 

Sirault 

60 

196 

3 

MIX 

H,F,H 

4,13,17 

1,12,1 

45 

D,2,6 

Sirault 

60 

197 

3 

MIX 

H,F,H 

6,14,17 

1,11,12 

45,90,90 

D ,  2 , 6 

Sirault 

60 

198 

3 

MIX 

H,F,F 

5,13,12 

2,1,12 

90,45,0 

2,6,4 

Simult 

60 

199 

3 

MIX 

H,F,F 

6,11,11 

12,11,1 

45,45,0 

C,  1 , 5 

Simult 

60 

200 

3 

MIX 

H,F,F 

4,14,12 

10,11,12 

90,45,45 

A, 2, 3 

Sirault 

60 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H 

H 


• 

AIRCRAFT  MODEL  TYPES 

FRIENDLY 

HOSTILE 

FW : 

FW: 

1  =A7 

4=MiG27 

2  =  Al0 

5=S  ul7 

3  =  F  16 

6=Su25 

RW : 

RW: 

1 1 =AH64 

15=Mi3 

1 2=UHl 

16=Mi24 

13=UH60 

17=Mi28 

14=CH3 

18=Mi?  (Hokum) 

FW  AIRCRAFT  PATTERNS 

A  —  90  degree  crossing 

pattern  commencing  at  10:00  azimuth 

3  —  45  degree  diagonal 

pattern  commencing  at  11:00  azimuth 

C  —  0  degree  ingress  pattern  commencing  at  12:00  azimuth  | 

D  —  45  degree  diagonal 

pattern  commencing  at  1:00  azimuth 

E  —  90  degree  crossing 

pattern  commencing  at  2:00  azimuth 

RW  AIRCRAFT  STANDS 

1  —  3  kilometer 

target  at  11:00  azimuth 

2  —  5  kilometer 

target  at  11:00  azimuth 

3  —  3  kilometer 

target  at  12:00  azimuth 

4  —  5  kilometer 

target  at  12:00  azimuth 

5  —  3  kilometer 

target  at  1:00  azimuth 

6  —  5  kilometer 

target  at  1:00  azimuth 

1  2  | 

4  |  6  |  5  km 

1  1  1 

3  |  5  j  3  km 

11:00 

12:00  1:00 
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SCENARIO  DIFFICULTY  RATINGS 


o  Extra  High  Difficulty  (XH).  Refers  to  scenarios  having  2  or 
;nore  targets  flying  tactical  maneuvers. 

o  High  Difficulty  (H).  FW  aircraft,  due  to  their  speed, 
maneuverability,  range,  and  altitude  present  a  more 
difficult  adversary  than  their  RW  counterparts,  especially 
when  they  ingress  at  zero  aspect.  A  FW  target  presented 
simultaneously  with  multiple  RW  threats  further  taxes  the 
soldiers'  abilities.  Mixed  ( FW  and  RW )  scenarios  thus 
represent  a  high  degree  of  difficulty.  Scenarios  presenting 
multiple  FW  threats  would  likely  be  one  or  two  levels  of 
difficulty  higher  than  this  (XH). 

o  Medium  High  Difficulty  (MH).  Based  on  the  definition  of  high 
difficulty  given  above,  the  next  difficulty  level  reflects 
single  FW  targets  presented  at  0  to  45  degrees  aspect. 

o  Medium  Difficulty  (M).  This  level  reflects  single,  crossing 
FW  scenarios  at  90  degrees  aspect.  Also  included  are 
triple-simultaneous  RW  scenarios.  The  multiple  RW  threat 
makes  for  a  challenging  scenario  in  terms  of  soldier 
workload . 

o  Medium  Low  Difficulty  (ML).  This  level  represents  the  double- 
simultaneous  RW  scenario,  and  single  or  double-sequential  RW 
scenarios  presenting  targets  at  maximum  ranges  or  with  small 
profiles  (zero  aspect).  Moderate  workload  or  moderate 
target  visibility  help  to  distinguish  this  from  the  lowest 
difficulty  level. 

o  Low  Difficulty  (L).  RW  targets  that  are  close  in  range  or 
present  a  side  view  orientation  are  easy  to  detect  and 
identify,  and  are  rapidly  engaged.  Therefore,  single,  or 
double-sequential  RW  scenarios  appear  to  be  the  easiest' 
ones . 
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APPENDIX  B 


PROCEDURES  FOR  ASSESSING 
SCENARIO  DIFFICULTY 


Scenario  Difficulty  Weighting  Procedure 

1.  A  list  was  generated  consisting  of  14  difficulty  factors 
having  subfactors  within  each  factor. 

2.  Five  Subject  Matter  Experts  (SME)  weighted  each  factor  for 
difficulty  on  a  scale  of  from  1  to  100  (l=easiest, 

100=hardest ) .  The  sum  of  the  14  factor  weights  always 
equaled  100. 

3.  Each  of  the  14  factors  contained  subfactors  (e.g.,  Model  Type 
factor  contained  a  subfactor  for  each  aircraft  model  being 
used).  Each  SME  rated  each  subfactor  for  difficulty  on  a 
scale  of  1  to  100  (subfactor  values  did  not  have  to  sum  to 
100  )  . 

4.  20  scenarios  were  developed.  Each  scenario  was  weighted 
using  the  factor  and  subfactor  scores  from  each  SME. 

Scenario  difficulty  scores  represented  subfactor  weights 
multiplied  by  associated  factor  weights  and  summed  over  all 
14  factors.  Thus,  a  point  total  existed  for  each  of  the  20 
scenarios  for  each  SME. 

5.  For  each  SME,  raw  scenario  scores  were  transformed  to 
standard  scores  of  1  to  5  (l=low,  2=medium-low,  3=medium, 
4=medi urn- hig h,  5=high)  using  the  following  procedure: 

a.  Scenario  scores  were  transformed  into  proportion 
scores  by  dividing  each  weight  by  the  largest 
weighting  score  given  by  that  specific  SME. 

(Example:  If  an  SME  rated  3  scenarios  50,  55,  65, 

then  each  scenario  would  be  turned  into  a 
proportion  score  with  65  as  the  denominator. 

50/65=. 77,  55/65=. 85,  65/65=1.00.)  Thus,  each 
weighting  score  was  turned  into  a  proportion  score 
with  each  base  being  that  SME's  highest  rating, 
thereby  controlling  for  differences  between  raters 
in  highest  score  given  and  range  of  scores. 

b.  Proportion  scores  ranged  from  lowest  to  1.00  across 
all  the  20  scenarios.  This  total  range  for  each 
SME  was  divided  into  5  equal-sized  categories.  (In 
the  example  above:  Range  was  1.00  minus  0.77=0.23. 

This  .23  range  was  divided  into  5  parts;  .770-. 816, 
.816-. 862,  .862-. 908,  .908-. 954,  .954-1.00.)  These 
5  equal-sized  categories  were  given  the  numbers  1 
to  5.  (Lowest  category=l,  highest  category=5) 
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5.  SME ' s  ratings  (now  labelled  1-5)  were  then  summed  and 

averaged  for  each  of  the  20  scenarios.  That  is,  the  mean  SME 
rating  was  determined  for  each  scenario.  Each  scenario  was 
given  a  final  label  of  L,  ML/  M,  MH ,  or  H  which  corresponded 
to  the  mean  SME  rating.  This  is  how  the  SME  weightings 
became  difficulty  indices  for  each  of  the  20  scenarios. 


Factors 

Criterion  1:  Target  Type  ( FW  or  RW)  —  RW  targets  are  generally 
easier  to  detect  and  identify  than  FW  targets  because  they 
are  usually  closer  in  range  upon  initial  1 ine-of-sight /  and 
do  not  roll,  pitch,  or  yaw. 

Criterion  2:  Target  Size  —  Target  model  types  vary  from  small 
( C H 1 )  to  large  (MIS).  The  larger  models  are  easier  to  see, 
and  therefore  to  detect  and  identify. 

Criterion  3:  Target  Model  —  Soldiers  are  typically  more  familiar 
with  some  aircraft  model  types  than  others.  For  example, 
soldiers  are  better  at  identifying  the  Hind-D  than  the 
Havoc . 

Criterion  4:  Target  Range  —  Obviously,  the  farther  away  the 
target  appears,  the  harder  it  is  to  detect  and  identify. 

This  variable  is  especially  relevant  for  RW  targets  as  they 
do  not  vary  in  range  once  exposed.  Thus,  the  farther  away 
they  are  presented,  the  higher  the  LOD  should  be.  For  FW, 
the  target  will  almost  always  begin  its  approach  from  beyond 
visible  range  when  ingressing. 

Criterion  5:  Target  Aspect  and  Offset  —  Targets  with  side  view 
orient-et  ions  are  easier  to  detect  and  to  identify  than  face 
view  (i.e.,  head-on)  orientations  because  the  target 
subtends  a  larger  visual  angle,  and  because  more  target 
features  are  visible.  Further,  the  farther  the  target  is 
from  the  fire  unit's  primary  target  line  or  from  a  cued 
azimuth,  the  longer  it  will  take  to  detect  it. 

Criterion  6:  Target  Altitude  —  Aircraft  flying  nap-of-the-earth 
or  at  extremely  high  altitudes  are  more  difficult  to  see 
than  those  flying  at  moderate  altitudes.  Further,  low 
targets  are  easier  to  see  as  the  elevation  above  the  terrain 
mask  increases. 

Criterion  7 :  Target  Speed  and  Maneuverability  —  Aircraft  flying 
extremely  fast  will  be  harder  to  see  and  will  present 
themselves  for  a  shorter  period  of  time  than  those  flying  at 
slower  speeds.  Further,  maneuvering  (dynamic)  targets  will 
be  harder  to  detect  and  engage  than  static  ones. 
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Criterion  8:  Target  Intent  (Friendly  or  Hostile)  —  RADES 

research  has  demonstrated  that  air  defenders  are  typically 
faster  and  more  accurate  in  responding  to  hostile  targets 
than  friendly  targets.  They  tend  to  adopt  "hostile 
expectancies"  whereby  the  target  default  is  hostile  when 
there  is  doubt  about  its  intent. 

Criterion  9:  Visibility  and  Contrast  Conditions  --  While  the 
typical  simulation  environment  will  have  clear  weather, 
daylight,  sky  background,  and  non-obscured  viewing 
conditions,  this  will  not  always  be  the  case  in  real  life 
situations.  In  order  to  generalize  a  scenario  to  other 
viewing  situations,  there  must  be  some  metric  to  gauge  the 
extent  to  which  reduced  visibility  will  increase  difficulty 
level.  Visibility  can  be  affected  by  atmospheric  conditions 
(e.g.  rain,  etc.),  windspeed,  cloudiness,  battlefield 
obscurants,  etc. 

Criterion  Id:  Terrain  Conditions  --  While  the  typical  simulation 
environment  will  be  the  desert  environment  with  a  sky  target 
background,  such  is  not  always  the  case  in  the  real  world. 
Treater  difficulty  would  be  expected  for  environments  having 
more  dense  terrain,  terrain  target  occulting  (obscuration), 
or  a  lower  contrast  ratio  between  target  and  background. 

Criterion  11:  Weather  Conditions  —  It  is  well  known  that 

performance  will  drop  as  a  result  of  extreme  temperatures  or 
weather  conditions.  This  criterion  relates  only  to  the 
effect  of  weather  on  the  soldier's  physical  abilities,  and 
not  on  visibility  which  was  covered  previously. 

Criterion  12:  Humber  of  Targets  —  More  than  one  target  can 

improve  detection  time  since  there  is  a  greater  likelihood 
that  a  target  will  appear  in  the  observer's  f ield-of-view. 
However,  multiple  targets  may  also  create  confusion  or  panic 
in  the  other  engagement  tasks  since  it  becomes  more 
difficult  to  sort  out  the  friends  from  the  foes,  and  engage 
the  target  posing  the  greatest  threat.  Therefore,  multiple 
target  scenarios  are  usually  associated  with  greater 
difficulty. 

Criterion  13:  Saturation  Level  —  This  criterion  relates 

primarily  to  workload  level  as  influenced  by  battlefield 
situations.  The  more  fatigued,  tired,  or  inattentive  the 
soldier  is,  the  poorer  his  performance  will  be.  A  fire  unit 
that  has  reacted  to  30  scenarios  will  likely  be  more  tired 
than  one  that  has  only  responded  to  3  scenarios  during  the 
same  time  frame;  but  tne  level  of  expectancy  for  that  fire 
unit  vill  likely  be  greater  as  well,  while  the  latter  fire 
unit  may  be  less  attentive.  Generally,  higher  saturation 
level  is  equated  to  greater  difficulty. 
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Criterion  14:  C3I  Conditions  --  The  use  of  doctrine,  tactics,  and 
C3I  vary  from  one  scenario  to  another  and  can  cause  drastic 
effects  on  performance,  especially  if  the  relevant 
information  is  not  easily  interpretable  or  if  it  is  untimely 
or  inaccurate.  Combinations  of  message  traffic  or  alerting 
and  cuing  updates  can  be  consistent  or  conflicting;  the  more 
conflicting  the  inputs,  the  more  confusion  that  ensues, 
resulting  in  either  hesitation  or  panic  on  the  part  of  the 
fire  unit.  For  example,  inputs  such  as  WCS  free,  air 
defense  warning  red,  and  IFF  return  unknown  are  all 
consistent  in  suggesting  that  an  approaching  aircraft  is 
hostile.  This  would  substant ial ly  lower  the  difficulty 
level. 


Subfactors 

Criterion  1:  Target  Type 
F W,  RW,  or  Mixed 

Criterion  2:  Target  Size 
Small  ( UHl  or  A7 ) 

Medium  (AH64  or  410) 

Large  (MI 24  or  3U17  ) 

Extra  Large  (MI  3  or  3U24) 

Criterion  3:  Target  Model 

?W:  A 7  (Corsair),  Al0  (Thunderbolt),  Fl6  (Fighting 

Falcon),  Fill,  3U7  (Fitter),  SU17/20/22  (Fitter),  SU24 
(Fencer),  3U25  (Frogfoot),  MIG27  (Flogger) 

RW:  UHl  (Iroquois),  UH60  (Blackhawk),  AHl  (Cobra),  AH64 

(Apache),  CH3  (Green  Giant),  MI8  (Hip),  MI24  (Hind), 

M 1 28  (Havoc),  MI?  (Hokum) 

Criterion  4:  Target  Range 

1-2  km,  3-4  km,  5-6  km,  7-8  km,  9-10  km,  11-14  km,  15-20  km 

Criterion  5:  target  Aspect 
90  degrees  (side  view) 

60  degrees  (side -tail  view) 

60  degrees  (side-face  view) 

30  degrees  (side-tail  view) 

30  degrees  (side-face  view) 

0  degrees  (tail  view) 

0  degrees  (face  view) 

Criterion  6:  Target  Altitude 


0  -  1 

degrees 

above 

horizon 

1.5-  3 

degrees 

above 

horizon 

3.5-  5 

degrees 

above 

horizon 

6  -  10 

degrees 

above 

hor izon 

11  -  15 

degrees 

above 

horizon 

16  -  25 

degrees 

above 

hor i z  on 

B4 


Criterion  7:  Target  Speed  (based  on  1/7  scale  aircraft) 
0-10  Mph  ( RW  hover) 

43  -  70  Mph  ( RW  maneuver) 

93  -  120  Mph  ( FW  maneuver) 

123  -  183  Mph  (FW  flyby) 

Criterion  8:  Target  Intent 
Friendly  or  Hostile 

Criterion  9:  Visibility  and  Contrast  Conditions 

1-2  km,  3-5  km,  6-10  km,  11-20  km,  21-40  km,  41+  km 
Clear  Sky,  Partly  Cloudy,  Overcast 

Criterion  13:  Terrain  Conditions 
Desert,  Forest,  Jungle 
Sky  Background,  Terrain  Background 

Criterion  11:  Weather  Conditions 
-10  to  20  degrees  F 
21  --  40  degrees  F 
41  —  60  degrees  F 
61  —  83  degrees  F 
31  — 100  degrees  F 
101+  degrees  F 

Criterion  12:  Number  of  Targets 
1,  2,  3,  4,  or  5 

Criterion  13:  Saturation  Level 
Scenarios  per  day: 

1  -  2  Scenarios  per  day 

3-9  Scenarios  per  day 

10  -  24  Scenarios  per  day 

25  -  40  Scenarios  per  day 

41+  Scenarios  per  day 

Arousal  Level : 

Fresh,  Average,  Fatigued 

Criterion  14:  C^I  Conditions 
Alert,  WCS ,  IFF: 

Red,  Free 
Red,  Tight 

Red,  Free,  &  IFF  Unknown 
Red,  Free,  &  IFF  Possible  Friend 
Red,  Tight,  &  IFF  Unknown 
Red,  Tight,  4  IFF  True  Friend 
Alerting  and  Cuing 
Alert  Once  per  Day 
Alert  Once  per  Trial 

Alert  4  Cue  (+  /  -15  degrees  accuracy) 

Alert  &  Cue  (+  /  -5  degrees  accuracy) 
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APPENDIX  C 


DESCRIPTIVE  STATISTICS  ON  20  STANDAPD  SCENARIOS 


Table  Cl 

Scenario  Descriptive  Statistics  ( Range=kilometers ;  Time=seconds; 
Pr oportion=percent ;  N=number  of  teams) 

SCENARIO  1  DESCRIPTIVES  SCENARIO  2  DESCRIPTIVES 


TARGET 

VARIABLE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

N 

1 ( A10  ) 

RDET 

12.2 

3.3 

6 

1 ( F16  ) 

RDET 

10.5 

2.0 

6 

RIFF 

10.0 

3.7 

4 

RIFF 

9.6 

2.5 

6 

RID 

7.3 

2.5 

6 

RID 

7.8 

2.5 

5 

TID 

17.7 

8.4 

6 

TID 

14.8 

4.5 

5 

RACQ 

7.5 

0.0 

1 

RACQ 

8.5 

0.9 

5 

IDCOR 

0.83 

0.41 

6 

RLOCK 

6.6 

0.8 

3 

FIRED 

0 . 00 

0 . 00 

6 

RFIRE 

7.0 

0.5 

2 

FRAT 

0.00 

— 

6 

TTOT 

25.5 

16.3 

2 

IDCOR 

0.50 

0.55 

6 

FIRED 

0.33 

0.52 

6 

EFFECT 

0.00 

0.00 

2 

FRAT 

0 . 00 

— 

6 

SCENARIO 

'  3  DESCRIPTIVES 

SCENARIO  4  DESCRIPTIVES 

TARGET 

VARIABLE 

MEAN 

SD 

N 

TARGET  VARIABLE 

MEAN 

SD 

N 

1 (Fill  ) 

RDET 

7.6 

0.7 

7 

1 (MIG27 ) RDET 

10.7 

2.8 

7 

RIFF 

6.9 

0.7 

7 

RIFF 

8.6 

4.0 

7 

RID 

5.7 

0.3 

7 

RID 

4.2 

0.7 

7 

TID 

12.4 

4.1 

7 

TID 

18.1 

6.5 

7 

RACQ 

6.0 

0.4 

4 

RACQ 

4.6 

1.5 

7 

RLOCK 

5.2 

0.0 

2 

RLOCK 

3.1 

1.0 

6 

RFIRE 

5.2 

0.0 

1 

RFIRE 

2.6 

1.1 

6 

TTOT 

26.0 

0.0 

1 

TTOT 

24.3 

5.0 

6 

IDCOR 

0.71 

0.49 

7 

IDCOR 

0.86 

0.38 

7 

FIRED 

0.14 

0.38 

7 

FIRED 

0.86 

0.38 

7 

EFFECT 

0 . 00 

0.00 

1 

EFFECT 

1.00 

0 . 00 

6 

FRAT 

0 . 00 

— 

7 

ATTRIT 

0.86 

— 

7 

ORD 

1.00 

0 . 00 

7 

Cl 


Table  Cl  (Continued) 
Scenario  Descriptives 


SCENARIO  5  DESCRIPTIVES 


SCENARIO  6  DESCRIPTIVES 


TARGET  VARIABLE 

MEAN 

SD 

N 

TARGET  VARIABLE 

MEAN 

SD 

N 

1 ( SO 2 5  )  RDET 

11.6 

2.9 

3 

1 ( SU24  )  RDET 

7.8 

0.7 

7 

RIFF 

8.7 

0.3 

3 

RIFF 

7.3 

0.5 

7 

RID 

7.2 

0.8 

2 

RID 

5.8 

0.3 

7 

TID 

18.0 

8.5 

2 

TID 

13.6 

3.3 

7 

RACQ 

5.5 

1.2 

3 

RACQ 

6.2 

0.4 

7 

RLOCK 

5.2 

0.0 

1 

RLOCK 

5.8 

0.4 

7 

RFIRE 

4.8 

0.1 

2 

RFIRE 

5.9 

0.6 

7 

TTOT 

28.0 

11.3 

2 

TTOT 

19.7 

3.3 

7 

IDCOR 

0.67 

0.58 

3 

IDCOR 

0.67 

0.58 

3 

FIRED 

0.67 

0.58 

3 

FIRED 

1.00 

0.00 

7 

EFFECT 

0.50 

0.71 

2 

EFFECT 

0.57 

0.53 

7 

ATTRIT 

0.33 

— 

3 

ATTRIT 

0.57 

— 

7 

ORD 

1.00 

— 

3 

ORD 

0.00 

0.00 

7 

SCENARIO 

'  7  DESCRIPTIVES 

SCENARIO  8  DESCRIPTIVES 

m  w  O  r*  nm 

VARIABLE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

N 

1 ( UHS0  ) 

TDET 

3.4 

1.1 

7 

1 ( CH  3  ) 

TDET 

2.7 

2.0 

7 

TIFF 

3.0 

2.8 

7 

TIFF 

5.8 

6.8 

5 

TID 

8.7 

4.2 

7 

TID 

4.1 

2.0 

7 

TACQ 

5.3 

1.6 

6 

TACQ 

3.7 

1.9 

4 

TLOCK 

2.7 

0.6 

3 

TLOCK 

2.7 

1.2 

3 

TFIRE 

3.0 

0.0 

1 

TFIRE 

3.0 

0.0 

2 

TTOT 

10.0 

0.0 

1 

TTOT 

9.0 

1.0 

3 

IDCOR 

0.57 

0.53 

7 

THAND 

5.3 

1.5 

3 

FIRED 

0.14 

0.38 

7 

IDCOR 

0.29 

0.49 

7 

EFFECT 

1.00 

0.00 

1 

FIRED 

0.43 

0.53 

7 

FRAT 

0.14 

— 

7 

EFFECT 

0.67 

0.58 

3 

FRAT 

0.29 

— 

7 

CP 


Table  Cl  (Continued) 

Scenario  Descriptives 

SCENARIO  9  DESCRIPTIVES  SCENARIO  10  DESCRIPTIVES 


'ARGET  VARIABLE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

N 

.  (MI?)  TDET 

4.5 

3.6 

3 

1 (MI8) 

TDET 

5.5 

6.8 

8 

TIFF 

1.5 

1.0 

7 

TIFF 

1.7 

1.1 

7 

TID 

5.7 

3.0 

8 

TID 

5.2 

3.0 

8 

TACO 

5.0 

2.2 

7 

TACQ 

4.9 

2.0 

8 

TLOCK 

2.6 

1.0 

7 

TLOCK 

2.0 

0.6 

7 

TFIRE 

3.6 

3.4 

7 

TFIRE 

2.3 

0.8 

6 

TTOT 

11.1 

4.3 

7 

TTOT 

9.1 

2.9 

6 

THANO 

6.5 

4.8 

6 

THAND 

4.6 

1.8 

5 

IDCOR 

0.87 

0.35 

8 

IDCOR 

0.87 

0.35 

8 

FIRED 

0.87 

0.35 

8 

FIRED 

0.75 

0 . 46 

8 

EFFECT 

0.86 

0.38 

7 

EFFECT 

0.67 

0.51 

6 

FRAT 

0.75 

— 

3 

FRAT 

0.50 

— 

8 

ORD 

0.62 

0.52 

8 

2 ( MI  8  ) 

TDET 

22.7 

1.2 

6 

TIFF 

2.2 

1.9 

5 

TID 

4.7 

1.9 

6 

TACQ 

5.4 

2.1 

3 

TLOCK 

2.3 

2.1 

3 

TFIRE 

2.0 

0.0 

3 

TTOT 

9.7 

2.2 

4 

THAND 

4.7 

1.5 

4 

IDCOR 

0.75 

0.46 

8 

FIRED 

0.50 

0.53 

8 

EFFECT 

0.25 

0.50 

4 

ATTRIT 

0.12 

— 

8 

ORD 

0.87 

0.35 

8 

C3 


Table  Cl  (Continued) 

Scenario  Descriptives 

SCENARIO  11  DESCRIPTIVES  SCENARIO  12  DESCRIPTIVES 


TARGET 

VARIABLE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

N 

1 ( MI  24  ) 

TDET 

3.6 

1.4 

7 

1 ( AH  1  ) 

TDET 

4.0 

1.9 

8 

TIFF 

8.9 

6.9 

7 

TIFF 

1.9 

1.1 

8 

TID 

5.0 

1.8 

6 

TID 

6.2 

2.8 

8 

TACQ 

4.6 

3.0 

5 

TACQ 

5.6 

1.7 

5 

TLOCK 

3.0 

1.0 

4 

TLOCK 

5.0 

2.8 

2 

TFIRE 

2.3 

0.6 

3 

TFIRE 

2.0 

0.0 

1 

TTOT 

9.0 

2.0 

3 

TTOT 

11.0 

0.0 

1 

THAND 

3.7 

3.2 

3 

THAND 

7.0 

0.0 

1 

IDCOR 

0.57 

0.53 

7 

IDCOR 

0.87 

0.35 

8 

FIRED 

3.43 

0.53 

7 

FIRED 

0.12 

0.35 

8 

EFFECT 

1.00 

0.00 

3 

EFFECT 

1.00 

0 . 00 

1 

ATT R I 7 

3.43 

— 

7 

FRAT 

0.12 

— 

3 

2 (MI  24  ) 

TDET 

25.2 

3.3 

5 

2  ( tJHl  ) 

TDET 

31 . 1 

7.7 

8 

TIFF 

2.7 

2.9 

4 

TIFF 

3.7 

4.2 

8 

TID 

2.2 

1.3 

5 

TID 

4.7 

2.3 

8 

TACQ 

3.4 

2.1 

5 

TACQ 

2.8 

1.5 

6 

TLOCK 

2.0 

0.8 

4 

TLOCK 

3.7 

3.1 

3 

TFIRE 

1.7 

1.0 

4 

TFIRE 

1.0 

0.0 

2 

TTOT 

6.4 

3.6 

5 

TTOT 

6.7 

2.1 

3 

THAND 

4.2 

2.4 

5 

THAND 

1.5 

0.7 

2 

IDCOR 

3.71 

0.49 

7 

IDCOR 

0.37 

0.52 

8 

FIRED 

0.71 

0.49 

7 

FIRED 

0.37 

0.52 

8 

EFFECT 

0.00 

0 . 00 

5 

EFFECT 

0.67 

0.58 

3 

ATT R IT 

0.0 

— 

7 

FRAT 

0.25 

— 

3 

ORD 

0.86 

0.38 

7 

< 


Table  Cl  (Continued) 
Scenario  Descriptives 

SCENARIO  13  DESCRIPTIVES 


SCENARIO  14  DESCRIPTIVES 


TARGET 

VARIABLE 

MEAN 

SD 

N 

iCiis) 

TDET 

5.7 

3.4 

8 

TIFF 

9.3 

8.5 

7 

TID 

6 . 6 

2.9 

8 

TACQ 

5.5 

4.4 

8 

TLOCK 

6.1 

9.7 

8 

TFIRE 

2.4 

1.3 

7 

TTOT 

11.4 

3.4 

7 

THAND 

5.0 

3.3 

7 

IDCOR 

0.87 

0.35 

8 

FIRED 

0.87 

0.35 

8 

EFFECT 

0.43 

0.53 

7 

ATTRIT 

0.37 

— 

3 

ORD 

1.00 

0.00 

8 

2 (MI  24  ) 

TDET 

33.5 

3.0 

6 

TIFF 

0.8 

1.0 

6 

TID 

3.0 

3.2 

6 

TACQ 

6.0 

5.2 

5 

TLOCK 

2.2 

0.5 

4 

TFIRE 

3.2 

2.2 

4 

TTOT 

12.5 

5 . 5 

4 

THAND 

5.0 

4.5 

4 

IDCOR 

0.37 

0.52 

3 

FIRED 

0.50 

0.53 

8 

EFFECT 

0.50 

0.57 

4 

ATTRIT 

0.25 

— 

3 

ORD 

1.00 

0.00 

8 

TARGET 

VARIABLE 

MEAN 

SD 

N 

1  (  CH  3 

TDET 

2.5 

1.0 

7 

or 

TIFF 

3.9 

4.7 

7 

MI  8) 

TID 

6.0 

1.6 

7 

TACQ 

5.7 

1.5 

6 

TLOCK 

3.2 

2.5 

6 

TFIRE 

2.3 

1.5 

6 

TTOT 

11.2 

2.7 

6 

THAND 

5.0 

1.8 

6 

IDCOR 

0.14 

0.38 

7 

FIRED 

0.86 

0.38 

7 

EFFECT 

1.00 

0.00 

6 

2  ( CH3 

TDET 

20.4 

7.5 

7 

or 

TIFF 

3.6 

5.5 

7 

MI  8  ) 

TID 

4.9 

2.3 

7 

TACQ 

7.4 

6.0 

7 

TLOCK 

3.9 

2.3 

7 

TFIRE 

1.7 

0.5 

7 

TTOT 

13.0 

5.4 

7 

THAND 

8.1 

5.8 

7 

IDCOR 

1.00 

0.00 

7 

FIRED 

1.00 

0 . 00 

7 

EFFECT 

0.71 

0.49 

7 

CH  3 

IDCOR 

0.14 

0.38 

7 

FIRED 

0.86 

0.38 

7 

EFFECT 

1.00 

0 . 00 

6 

FRAT 

0.86 

0.38 

7 

MI  8 

IDCOR 

1.00 

0.00 

7 

FIRED 

1.00 

0 . 00 

7 

EFFECT 

0.71 

0.49 

7 

ATTRIT 

0.71 

— 

7 

ORD 

1.00 

0 . 00 

7 

C5 


Table  Cl  (Continued) 
Scenario  Descriptives 


SCENARIO  15  DESCRIPTIVES  SCENARIO  16  DESCRIPTIVES 


TARGET 

VARIABLE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

1 

1  (MI  8 

TDET 

3.1 

1.3 

7 

1  ( AHl 

TDET 

8.9 

12.9 

7 

or 

TIFF 

3.6 

5.1 

5 

or 

TIFF 

3.5 

4.4 

6 

MI  28 

TID 

4.7 

1.9 

7 

MI  28 

TID 

5.3 

3.7 

6 

or 

TACQ 

3.9 

2.5 

7 

or 

TACQ 

7.3 

2.1 

3 

MI 24  ) 

TLOCK 

3.2 

1.5 

6 

MI  24  ) 

TLOCK 

3.8 

1.1 

5 

TFIRE 

2.2 

0.5 

5 

TFIRE 

4.2 

6.0 

4 

TTOT 

8.8 

1.3 

6 

TTOT 

8.7 

5.2 

4 

THAND 

4.3 

0.8 

6 

THAND 

6.0 

6.4 

5 

IDCOR 

0.86 

0.38 

7 

IDCOR 

0.71 

0.49 

7 

FIRED 

0.71 

0.49 

7 

FIRED 

0.71 

0.49 

7 

EFFECT 

0.80 

0.45 

5 

EFFECT 

0.60 

0.54 

5 

2  ( MI  8 

TDET 

17.6 

2.0 

7 

2  ( AHl 

TDET 

18.6 

7.5 

5 

or 

TIFF 

0.8 

1.0 

6 

or 

TIFF 

4.0 

4.1 

5 

MI28 

TID 

4.2 

1.2 

6 

MI  28 

TID 

6.4 

2.2 

5 

or 

TACQ 

4.7 

1.5 

6 

or 

TACQ 

3.2 

2.2 

5 

MI 24  ) 

TLOCK 

2.5 

1.4 

6 

MI  24  ) 

TLOCK 

5.3 

3.1 

3 

TFIRE 

2.2 

0.8 

6 

TFIRE 

2.0 

1.0 

3 

TTOT 

9.3 

1.4 

6 

TTOT 

8.7 

3.8 

4 

THAND 

5.2 

0.8 

6 

THAND 

2.7 

3.2 

4 

IDCOR 

0.86 

0.38 

7 

IDCOR 

0.57 

0.53 

7 

FIRED 

0.86 

0.38 

7 

FIRED 

0.57 

0.53 

7 

EFFECT 

1.00 

0.00 

6 

EFFECT 

0.50 

0.58 

4 

3  (MI  8 

TDET 

30.7 

4.0 

7 

3  ( AHl 

TDET 

31.8 

8.3 

5 

or 

TIFF 

3.0 

2.8 

7 

or 

TIFF 

1.0 

1.0 

3 

MI  28 

TID 

8.4 

3.8 

7 

MI  28 

TID 

6.5 

1.3 

4 

or 

TACQ 

6.2 

4.2 

6 

or 

TACQ 

4.0 

2.7 

4 

MI 24  ) 

TLOCK 

3.4 

1.9 

5 

MI 24  ) 

TLOCK 

4.0 

0.0 

1 

TFIRE 

1.8 

0.4 

5 

TFIRE 

2.0 

0.0 

1 

TTOT 

12.8 

4.9 

5 

TTOT 

17.7 

10.0 

3 

THAND 

4.0 

1.9 

5 

THAND 

11.7 

10.8 

3 

IDCOR 

0.86 

0.38 

7 

IDCOR 

0.43 

0.53 

7 

FIRED 

0.71 

0.49 

7 

FIRED 

0.57 

0.53 

7 

EFFECT 

0.40 

0.55 

5 

EFFECT 

0.00 

0.00 

3 

MI  8 

IDCOR 

1.00 

0.00 

/ 

AHl 

IDCOR 

0.57 

0.53 

7 

FIRED 

1.00 

0.00 

7 

FIRED 

0.57 

0.53 

7 

EFFECT 

0.86 

0.38 

7 

EFFECT 

0.25 

0.50 

4 

ATTRIT 

0.86 

— 

7 

FRAT 

0.14 

— 

7 

ORD 

1.00 

0.00 

7 

MI  24 

IDCOR 

0.57 

0.53 

7 

MI  28 

IDCOR 

0.71 

0.49 

7 

FIRED 

0.71 

0.49 

7 

FIRED 

0.71 

0.49 

7 

EFFECT 

0.60 

0.55 

5 

EFFECT 

1.00 

0 . 00 

5 

ATTRIT 

0.43 

— 

7 

ATTRIT 

0.71 

— 

7 

ORD 

1.00 

0.00 

7 

ORD 

1.00 

0.00 

7 

MI  28 

IDCOR 

0.57 

0.53 

7 

MI  24 

IDCOR 

0.86 

0.38 

7 

FIRED 

0.43 

0.53 

7 

FTRF.n 

0.57 

0.53 

7 

EFFECT 

0.33 

0.58 

3 

EFFECT 

0.25 

0.50 

4 

ATTRIT 

0.14 

— 

7 

ATTRIT 

0.14 

— 

7 

ORD 

1.00 

0 . 00 

7 

ORD 

1.00 

0.00 

7 

C6 


Table  Cl  (Continued) 
Scenario  Descriptives 


SCENARIO  17  DESCRIPTIVES  SCENARIO  18  DESCRIPTIVES 


TARGET 

VARIABLE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

N 

1  (  CH3 

TDET 

1.3 

0.5 

7 

1  (  A7  ) 

RDET 

11.1 

2.2 

6 

or 

TIFF 

2.3 

2.4 

7 

RIFF 

7.5 

3.7 

5 

MI  28 

TID 

4.0 

1.8 

7 

RID 

5.0 

3.7 

6 

or 

TACQ 

4.4 

1.3 

5 

TID 

21.2 

11.1 

6 

UHl  ) 

TLOCK 

2.2 

1.6 

5 

RACQ 

5.5 

1.3 

3 

TFIRE 

1.8 

0.8 

5 

RLOCK 

4.4 

0.0 

1 

TTOT 

8.4 

1.8 

5 

RFIRE 

3.6 

0.1 

2 

THAND 

4.6 

2.3 

5 

TTOT 

29.0 

15.6 

2 

IDCOR 

0.81 

0.49 

7 

IDCOR 

0.83 

0.41 

6 

FIRED 

0.71 

0.49 

7 

FIRED 

0.33 

0.52 

6 

EFFECT 

1.00 

0.00 

5 

EFFECT 

0.50 

0.71 

2 

2  (  CH  3 

TDET 

13.1 

1.8 

7 

FRAT 

0.17 

— 

6 

or 

TIFF 

2.4 

2.9 

7 

2 (MI  24  ) 

TDET 

9.5 

4.9 

2 

MI  28 

TID 

4.1 

2.9 

7 

TIFF 

1.0 

0.0 

1 

or 

TACQ 

3.2 

2.2 

4 

TID 

3.0 

0.0 

1 

UHl ) 

TLOCK 

2.0 

1.4 

2 

TACQ 

5.0 

0.0 

1 

TFIRE 

2.0 

0.0 

3 

TLOCK 

2.0 

0.0 

2 

TTOT 

7.7 

1.5 

3 

TFIRE 

7.0 

"7  "» 

/  •  X 

2 

THAND 

5.0 

0.0 

3 

TTOT 

9.0 

0.0 

1 

IDCOR 

0.57 

0.53 

7 

THAND 

11.0 

7.1 

2 

FIRED 

0.43 

0.53 

7 

IDCOR 

0.33 

0.52 

6 

EFFECT 

0.67 

0.58 

3 

FIRED 

0.33 

0.52 

6 

3  (  CH  3 

TDET 

25.3 

4.9 

7 

EFFECT 

0.50 

0.71 

2 

or 

TIFF 

3.5 

6.7 

6 

ATTRIT 

0.17 

— 

6 

MI  28 

TID 

7.3 

5.3 

7 

ORD 

1.00 

0.00 

6 

or 

TACQ 

5.5 

1.3 

4 

UHl ) 

TLOCK 

3.0 

1.4 

2 

TFIRE 

3.5 

1.7 

4 

TTOT 

10.6 

3.2 

5 

THAND 

4.2 

2.1 

4 

IDCOR 

0.29 

0.49 

7 

FIRED 

0.71 

0.49 

7 

EFFECT 

0.20 

0.45 

5 

CH  3 

IDCOR 

0.14 

0.38 

7 

FIRED 

0.86 

0.38 

7 

EFFECT 

0.67 

0.52 

6 

FRAT 

0.57 

— 

7 

MI  28 

IDCOR 

0.71 

0.49 

7 

FIRED 

0.71 

0.49 

7 

EFFECT 

0.80 

0.45 

5 

ATTRIT 

0.57 

— 

7 

ORD 

0.57 

0.53 

7 

UHl 

IDCOR 

0.71 

0.49 

7 

FIRED 

0.29 

0.49 

7 

EFFECT 

0.00 

0.00 

2 

FRAT 

0.00 

0 . 00 

7 

07 


Table  Cl  (Continued) 
Scenario  Descriptives 


SCENARIO  19  DESCRIPTIVES  SCENARIO  20  DESCRIPTIVES 


TARGET 

VARIA3LE 

MEAN 

SD 

N 

TARGET 

VARIABLE 

MEAN 

SD 

N 

1  (MI? 

TDET 

3.8 

1.9 

5 

1 (SU25  ) 

RDET 

12.6 

2.0 

7 

or 

TIFF 

11.0 

13.7 

4 

RIFF 

10.5 

2.3 

6 

MI  28  ) 

TID 

7.8 

7.1 

5 

RID 

4.4 

2.4 

6 

TACQ 

8.7 

6.7 

4 

TID 

27.0 

10.3 

6 

TLOCK 

4.6 

3.1 

5 

RACQ 

6.4 

2.9 

6 

TFIRE 

5.7 

6.4 

4 

RLOCK 

3.4 

1.4 

5 

TTOT 

16.7 

7.7 

4 

RFIRE 

2.3 

0.7 

5 

THAND 

8.0 

9.4 

4 

TTOT 

34.8 

8.0 

5 

IDCOR 

0.80 

0.45 

5 

IDCOR 

0.71 

0.49 

7 

FIRED 

0.80 

0.45 

5 

FIRED 

0.71 

0.49 

7 

EFFECT 

1.00 

0 . 00 

4 

EFFECT 

1.00 

0 . 00 

5 

2  (MI? 

TDET 

30.2 

11.3 

6 

ATTRIT 

0.71 

— 

7 

or 

TIFF 

2.8 

3.6 

5 

ORD 

0.86 

0.38 

7 

MI  28  ) 

TID 

5.0 

3.9 

6 

2 ( UH  60 

TDET 

26.0 

11.1 

7 

TACQ 

4.0 

1.8 

6 

or 

TIFF 

8.4 

13.8 

7 

TLOCK 

8.0 

14.5 

5 

MI  8  ) 

TID 

6.0 

2.8 

6 

TFIRE 

2.0 

0.7 

5 

TACQ 

5.8 

5.0 

5 

TTOT 

14.2 

14.6 

5 

TLOCK 

3.6 

2.7 

5 

THAND 

9.6 

10.4 

5 

TFIRE 

7.6 

10.5 

5 

IDCOR 

0.67 

0.52 

6 

TTOT 

16.4 

7.6 

5 

FIRED 

0.83 

0.41 

6 

THAND 

9.5 

9.7 

4 

EFFECT 

0.75 

0.50 

4 

IDCOR 

0.71 

0.49 

7 

3 ( SU17  ) 

RDET 

14.6 

1.7 

2 

FIRED 

0.71 

0.49 

7 

RIFF 

7.7 

3.0 

2 

EFFECT 

0.50 

0.58 

4 

RID 

6.7 

4.2 

2 

UH  60 

IDCOR 

0.50 

0.58 

4 

TID 

26.5 

9.2 

2 

FIRED 

0.33 

0.52 

6 

RACQ 

6.1 

3.7 

2 

EFFECT 

1.00 

0.00 

2 

RLOCK 

5.3 

3.0 

2 

FRAT 

0.33 

— 

6 

RFIRE 

4.8 

2.6 

2 

MI  8 

IDCOR 

0.29 

0.49 

7 

TTOT 

36.0 

7.1 

2 

FIRED 

0.43 

0.53 

7 

IDCOR 

1.00 

0.00 

2 

EFFECT 

0 . 00 

0 . 00 

2 

FIRED 

1.00 

0 . 00 

2 

ATTRIT 

0.00 

— 

7 

1.00 

0 . 00 

2 

ORD 

1.00 

0.00 

7 

ATTRIT 

0.29 

— 

7 

ORD 

0.50 

0.71 

2 

MI? 

IDCOR 

0.36 

0.38 

7 

FIRED 

0.86 

0.38 

7 

EFFECT 

0.83 

0.41 

6 

ATTRIT 

0.71 

— 

7 

ORD 

1.00 

0.00 

7 

MI  28 

IDCOR 

0.57 

0.53 

7 

FIRED 

0.71 

0.49 

7 

EFFECT 

1.00 

0 . 00 

4 

ATTRIT 

0.57 

7 

ORD 

0.57 

0.53 

7 

NOTE :  No  engagement  data  on  target  number  3 


C8 


